[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260807 03:37:25.879840 16154 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.15.198.190:41955
I20260807 03:37:25.882483 16154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:37:25.884250 16154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:37:25.926568 16162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:37:25.928301 16167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:37:25.928294 16163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:37:25.933908 16154 server_base.cc:1034] running on GCE node
I20260807 03:37:27.209434 16154 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260807 03:37:27.209717 16154 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260807 03:37:27.209872 16154 hybrid_clock.cc:648] HybridClock initialized: now 1786073847209857 us; error 0 us; skew 500 ppm
I20260807 03:37:27.210997 16154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:37:27.217304 16154 webserver.cc:460] Webserver started at http://127.15.198.190:40855/ using document root <none> and password file <none>
I20260807 03:37:27.218616 16154 fs_manager.cc:359] Metadata directory not provided
I20260807 03:37:27.218806 16154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:37:27.219367 16154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:37:27.226985 16154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786073845599630-16154-0/minicluster-data/master-0-root/instance:
uuid: "044fc5f21a5d4b1597840dabbdca0f5a"
format_stamp: "Formatted at 2026-08-07 03:37:27 on dist-test-slave-7vjv"
I20260807 03:37:27.237211 16154 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.006s sys 0.001s
I20260807 03:37:27.243829 16179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:37:27.245445 16154 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.001s
I20260807 03:37:27.245808 16154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786073845599630-16154-0/minicluster-data/master-0-root
uuid: "044fc5f21a5d4b1597840dabbdca0f5a"
format_stamp: "Formatted at 2026-08-07 03:37:27 on dist-test-slave-7vjv"
I20260807 03:37:27.246156 16154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786073845599630-16154-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786073845599630-16154-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786073845599630-16154-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:37:27.295044 16154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:37:27.296837 16154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:37:27.297349 16154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:37:27.381412 16154 rpc_server.cc:307] RPC server started. Bound to: 127.15.198.190:41955
I20260807 03:37:27.381500 16262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.198.190:41955 every 8 connection(s)
I20260807 03:37:27.392936 16263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:27.421346 16263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 044fc5f21a5d4b1597840dabbdca0f5a: Bootstrap starting.
I20260807 03:37:27.429330 16263 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 044fc5f21a5d4b1597840dabbdca0f5a: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:27.431672 16263 log.cc:824] T 00000000000000000000000000000000 P 044fc5f21a5d4b1597840dabbdca0f5a: Log is configured to *not* fsync() on all Append() calls
I20260807 03:37:27.436821 16263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 044fc5f21a5d4b1597840dabbdca0f5a: No bootstrap required, opened a new log
I20260807 03:37:27.460680 16263 raft_consensus.cc:348] T 00000000000000000000000000000000 P 044fc5f21a5d4b1597840dabbdca0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "044fc5f21a5d4b1597840dabbdca0f5a" member_type: VOTER }
I20260807 03:37:27.461541 16263 raft_consensus.cc:374] T 00000000000000000000000000000000 P 044fc5f21a5d4b1597840dabbdca0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:27.461936 16263 raft_consensus.cc:729] T 00000000000000000000000000000000 P 044fc5f21a5d4b1597840dabbdca0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 044fc5f21a5d4b1597840dabbdca0f5a, State: Initialized, Role: FOLLOWER
I20260807 03:37:27.463327 16263 consensus_queue.cc:260] T 00000000000000000000000000000000 P 044fc5f21a5d4b1597840dabbdca0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "044fc5f21a5d4b1597840dabbdca0f5a" member_type: VOTER }
I20260807 03:37:27.463970 16263 raft_consensus.cc:388] T 00000000000000000000000000000000 P 044fc5f21a5d4b1597840dabbdca0f5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:37:27.464336 16263 raft_consensus.cc:482] T 00000000000000000000000000000000 P 044fc5f21a5d4b1597840dabbdca0f5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:37:27.464694 16263 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 044fc5f21a5d4b1597840dabbdca0f5a [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:37:27.469331 16263 raft_consensus.cc:504] T 00000000000000000000000000000000 P 044fc5f21a5d4b1597840dabbdca0f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "044fc5f21a5d4b1597840dabbdca0f5a" member_type: VOTER }
I20260807 03:37:27.470414 16263 leader_election.cc:302] T 00000000000000000000000000000000 P 044fc5f21a5d4b1597840dabbdca0f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 044fc5f21a5d4b1597840dabbdca0f5a; no voters:
I20260807 03:37:27.472512 16263 leader_election.cc:288] T 00000000000000000000000000000000 P 044fc5f21a5d4b1597840dabbdca0f5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:37:27.472910 16266 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 044fc5f21a5d4b1597840dabbdca0f5a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:37:27.476003 16266 raft_consensus.cc:686] T 00000000000000000000000000000000 P 044fc5f21a5d4b1597840dabbdca0f5a [term 1 LEADER]: Becoming Leader. State: Replica: 044fc5f21a5d4b1597840dabbdca0f5a, State: Running, Role: LEADER
I20260807 03:37:27.477170 16266 consensus_queue.cc:237] T 00000000000000000000000000000000 P 044fc5f21a5d4b1597840dabbdca0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "044fc5f21a5d4b1597840dabbdca0f5a" member_type: VOTER }
I20260807 03:37:27.478370 16263 sys_catalog.cc:564] T 00000000000000000000000000000000 P 044fc5f21a5d4b1597840dabbdca0f5a [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:37:27.493765 16267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 044fc5f21a5d4b1597840dabbdca0f5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 044fc5f21a5d4b1597840dabbdca0f5a. Latest consensus state: current_term: 1 leader_uuid: "044fc5f21a5d4b1597840dabbdca0f5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "044fc5f21a5d4b1597840dabbdca0f5a" member_type: VOTER } }
I20260807 03:37:27.494043 16269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 044fc5f21a5d4b1597840dabbdca0f5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "044fc5f21a5d4b1597840dabbdca0f5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "044fc5f21a5d4b1597840dabbdca0f5a" member_type: VOTER } }
I20260807 03:37:27.495255 16269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 044fc5f21a5d4b1597840dabbdca0f5a [sys.catalog]: This master's current role is: LEADER
I20260807 03:37:27.495223 16267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 044fc5f21a5d4b1597840dabbdca0f5a [sys.catalog]: This master's current role is: LEADER
I20260807 03:37:27.498312 16280 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:37:27.515265 16280 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:37:27.517386 16154 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260807 03:37:27.534076 16280 catalog_manager.cc:1347] Generated new cluster ID: 3dc2a5fbdb3545dda3342c3ed3fc29b1
I20260807 03:37:27.534338 16280 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:37:27.561118 16280 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:37:27.563027 16280 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:37:27.598186 16280 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 044fc5f21a5d4b1597840dabbdca0f5a: Generated new TSK 0
I20260807 03:37:27.599321 16280 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:37:27.649830 16154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:37:27.662334 16291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:37:27.664651 16294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:37:27.667177 16297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:37:27.668888 16154 server_base.cc:1034] running on GCE node
I20260807 03:37:27.669749 16154 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260807 03:37:27.669993 16154 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260807 03:37:27.670172 16154 hybrid_clock.cc:648] HybridClock initialized: now 1786073847670154 us; error 0 us; skew 500 ppm
I20260807 03:37:27.670749 16154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:37:27.675813 16154 webserver.cc:460] Webserver started at http://127.15.198.129:43061/ using document root <none> and password file <none>
I20260807 03:37:27.676448 16154 fs_manager.cc:359] Metadata directory not provided
I20260807 03:37:27.676676 16154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:37:27.676956 16154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:37:27.678269 16154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786073845599630-16154-0/minicluster-data/ts-0-root/instance:
uuid: "9ca3bab29b4c4dce908de0247c715423"
format_stamp: "Formatted at 2026-08-07 03:37:27 on dist-test-slave-7vjv"
I20260807 03:37:27.684692 16154 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260807 03:37:27.690464 16303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:37:27.691282 16154 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260807 03:37:27.691557 16154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786073845599630-16154-0/minicluster-data/ts-0-root
uuid: "9ca3bab29b4c4dce908de0247c715423"
format_stamp: "Formatted at 2026-08-07 03:37:27 on dist-test-slave-7vjv"
I20260807 03:37:27.691830 16154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786073845599630-16154-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786073845599630-16154-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786073845599630-16154-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:37:27.713836 16154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:37:27.715144 16154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:37:27.718452 16154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:37:27.724001 16154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:37:27.724313 16154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260807 03:37:27.724635 16154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:37:27.724785 16154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:37:27.841499 16154 rpc_server.cc:307] RPC server started. Bound to: 127.15.198.129:36375
I20260807 03:37:27.841797 16407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.198.129:36375 every 8 connection(s)
I20260807 03:37:27.849714 16154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:37:27.863273 16416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:37:27.864507 16417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:37:27.870357 16422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:37:27.876211 16154 server_base.cc:1034] running on GCE node
I20260807 03:37:27.877521 16154 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260807 03:37:27.877835 16154 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260807 03:37:27.878055 16154 hybrid_clock.cc:648] HybridClock initialized: now 1786073847878042 us; error 0 us; skew 500 ppm
I20260807 03:37:27.878890 16154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:37:27.883589 16409 heartbeater.cc:347] Connected to a master server at 127.15.198.190:41955
I20260807 03:37:27.884081 16409 heartbeater.cc:464] Registering TS with master...
I20260807 03:37:27.884192 16154 webserver.cc:460] Webserver started at http://127.15.198.130:44027/ using document root <none> and password file <none>
I20260807 03:37:27.884927 16154 fs_manager.cc:359] Metadata directory not provided
I20260807 03:37:27.885044 16409 heartbeater.cc:511] Master 127.15.198.190:41955 requested a full tablet report, sending...
I20260807 03:37:27.885188 16154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:37:27.885546 16154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:37:27.887060 16154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786073845599630-16154-0/minicluster-data/ts-1-root/instance:
uuid: "05aa1d36d28a48138040f6c78c4b34a5"
format_stamp: "Formatted at 2026-08-07 03:37:27 on dist-test-slave-7vjv"
I20260807 03:37:27.888674 16208 ts_manager.cc:194] Registered new tserver with Master: 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129:36375)
I20260807 03:37:27.891225 16208 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33160
I20260807 03:37:27.896368 16154 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.002s sys 0.009s
I20260807 03:37:27.901435 16428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:37:27.902278 16154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260807 03:37:27.902590 16154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786073845599630-16154-0/minicluster-data/ts-1-root
uuid: "05aa1d36d28a48138040f6c78c4b34a5"
format_stamp: "Formatted at 2026-08-07 03:37:27 on dist-test-slave-7vjv"
I20260807 03:37:27.902949 16154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786073845599630-16154-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786073845599630-16154-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786073845599630-16154-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:37:27.927291 16154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:37:27.928694 16154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:37:27.931113 16154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:37:27.933929 16154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:37:27.934140 16154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:37:27.934355 16154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:37:27.934500 16154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:37:28.041044 16154 rpc_server.cc:307] RPC server started. Bound to: 127.15.198.130:39359
I20260807 03:37:28.041174 16525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.198.130:39359 every 8 connection(s)
I20260807 03:37:28.048022 16154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:37:28.057078 16532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:37:28.060016 16535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:37:28.066695 16154 server_base.cc:1034] running on GCE node
I20260807 03:37:28.067198 16526 heartbeater.cc:347] Connected to a master server at 127.15.198.190:41955
I20260807 03:37:28.067589 16526 heartbeater.cc:464] Registering TS with master...
I20260807 03:37:28.068488 16526 heartbeater.cc:511] Master 127.15.198.190:41955 requested a full tablet report, sending...
W20260807 03:37:28.068931 16541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:37:28.070127 16154 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260807 03:37:28.070370 16154 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260807 03:37:28.070554 16154 hybrid_clock.cc:648] HybridClock initialized: now 1786073848070538 us; error 0 us; skew 500 ppm
I20260807 03:37:28.070680 16208 ts_manager.cc:194] Registered new tserver with Master: 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130:39359)
I20260807 03:37:28.071244 16154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:37:28.073267 16208 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33166
I20260807 03:37:28.074398 16154 webserver.cc:460] Webserver started at http://127.15.198.131:40701/ using document root <none> and password file <none>
I20260807 03:37:28.075116 16154 fs_manager.cc:359] Metadata directory not provided
I20260807 03:37:28.075376 16154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:37:28.075711 16154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:37:28.077052 16154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786073845599630-16154-0/minicluster-data/ts-2-root/instance:
uuid: "ec8c86cff51f48bf9ecc5177e6266b7b"
format_stamp: "Formatted at 2026-08-07 03:37:28 on dist-test-slave-7vjv"
I20260807 03:37:28.082118 16154 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260807 03:37:28.085620 16550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:37:28.086429 16154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 03:37:28.086711 16154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786073845599630-16154-0/minicluster-data/ts-2-root
uuid: "ec8c86cff51f48bf9ecc5177e6266b7b"
format_stamp: "Formatted at 2026-08-07 03:37:28 on dist-test-slave-7vjv"
I20260807 03:37:28.086964 16154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786073845599630-16154-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786073845599630-16154-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786073845599630-16154-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:37:28.103567 16154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:37:28.104717 16154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:37:28.106133 16154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:37:28.108289 16154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:37:28.108466 16154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:37:28.108742 16154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:37:28.108875 16154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:37:28.213814 16154 rpc_server.cc:307] RPC server started. Bound to: 127.15.198.131:37775
I20260807 03:37:28.214368 16644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.198.131:37775 every 8 connection(s)
I20260807 03:37:28.234313 16646 heartbeater.cc:347] Connected to a master server at 127.15.198.190:41955
I20260807 03:37:28.234732 16646 heartbeater.cc:464] Registering TS with master...
I20260807 03:37:28.235534 16646 heartbeater.cc:511] Master 127.15.198.190:41955 requested a full tablet report, sending...
I20260807 03:37:28.237461 16208 ts_manager.cc:194] Registered new tserver with Master: ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131:37775)
I20260807 03:37:28.237974 16154 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.016993504s
I20260807 03:37:28.239454 16208 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33172
I20260807 03:37:28.242300 16154 test_util.cc:268] Using random seed: -1186701226
I20260807 03:37:28.302585 16208 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33174:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 03:37:28.307338 16208 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 03:37:28.417543 16466 tablet_service.cc:1467] Processing CreateTablet for tablet f0191ce545e0490986331a9c562b056d (DEFAULT_TABLE table=test-workload [id=b66c94eff33b468d8b82a87be99491d0]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260807 03:37:28.422322 16464 tablet_service.cc:1467] Processing CreateTablet for tablet fe184e38bf0d4e51bf42238d161ac738 (DEFAULT_TABLE table=test-workload [id=b66c94eff33b468d8b82a87be99491d0]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260807 03:37:28.425627 16466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0191ce545e0490986331a9c562b056d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:28.424355 16460 tablet_service.cc:1467] Processing CreateTablet for tablet 10452db9b82745b68c6c353b25eff5ef (DEFAULT_TABLE table=test-workload [id=b66c94eff33b468d8b82a87be99491d0]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260807 03:37:28.426148 16463 tablet_service.cc:1467] Processing CreateTablet for tablet e9e0fd324ec84973b604e22e851d7b49 (DEFAULT_TABLE table=test-workload [id=b66c94eff33b468d8b82a87be99491d0]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260807 03:37:28.429349 16462 tablet_service.cc:1467] Processing CreateTablet for tablet 485e8d9a337d4e4f9207f737e3a8812b (DEFAULT_TABLE table=test-workload [id=b66c94eff33b468d8b82a87be99491d0]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260807 03:37:28.430377 16461 tablet_service.cc:1467] Processing CreateTablet for tablet d07d00a26d54438392d5db566a17627b (DEFAULT_TABLE table=test-workload [id=b66c94eff33b468d8b82a87be99491d0]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260807 03:37:28.430747 16457 tablet_service.cc:1467] Processing CreateTablet for tablet a4e0d750de56427e8a0af5bc030271c4 (DEFAULT_TABLE table=test-workload [id=b66c94eff33b468d8b82a87be99491d0]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260807 03:37:28.428380 16458 tablet_service.cc:1467] Processing CreateTablet for tablet 5e409bda14f142959c8127f3bd2643fc (DEFAULT_TABLE table=test-workload [id=b66c94eff33b468d8b82a87be99491d0]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260807 03:37:28.435638 16464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe184e38bf0d4e51bf42238d161ac738. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:28.435720 16456 tablet_service.cc:1467] Processing CreateTablet for tablet 242a724d826340b693e5b0767a1ba28c (DEFAULT_TABLE table=test-workload [id=b66c94eff33b468d8b82a87be99491d0]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260807 03:37:28.436486 16458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e409bda14f142959c8127f3bd2643fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:28.437166 16460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10452db9b82745b68c6c353b25eff5ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:28.437841 16461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d07d00a26d54438392d5db566a17627b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:28.438519 16463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9e0fd324ec84973b604e22e851d7b49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:28.439232 16456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 242a724d826340b693e5b0767a1ba28c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:28.439941 16457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4e0d750de56427e8a0af5bc030271c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:28.442350 16462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 485e8d9a337d4e4f9207f737e3a8812b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:28.455363 16595 tablet_service.cc:1467] Processing CreateTablet for tablet f0191ce545e0490986331a9c562b056d (DEFAULT_TABLE table=test-workload [id=b66c94eff33b468d8b82a87be99491d0]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260807 03:37:28.456200 16593 tablet_service.cc:1467] Processing CreateTablet for tablet e9e0fd324ec84973b604e22e851d7b49 (DEFAULT_TABLE table=test-workload [id=b66c94eff33b468d8b82a87be99491d0]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260807 03:37:28.457717 16593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9e0fd324ec84973b604e22e851d7b49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:28.457471 16594 tablet_service.cc:1467] Processing CreateTablet for tablet fe184e38bf0d4e51bf42238d161ac738 (DEFAULT_TABLE table=test-workload [id=b66c94eff33b468d8b82a87be99491d0]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260807 03:37:28.458781 16594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe184e38bf0d4e51bf42238d161ac738. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:28.460124 16592 tablet_service.cc:1467] Processing CreateTablet for tablet 485e8d9a337d4e4f9207f737e3a8812b (DEFAULT_TABLE table=test-workload [id=b66c94eff33b468d8b82a87be99491d0]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260807 03:37:28.461581 16592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 485e8d9a337d4e4f9207f737e3a8812b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:28.462929 16589 tablet_service.cc:1467] Processing CreateTablet for tablet 10452db9b82745b68c6c353b25eff5ef (DEFAULT_TABLE table=test-workload [id=b66c94eff33b468d8b82a87be99491d0]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260807 03:37:28.465384 16587 tablet_service.cc:1467] Processing CreateTablet for tablet 5e409bda14f142959c8127f3bd2643fc (DEFAULT_TABLE table=test-workload [id=b66c94eff33b468d8b82a87be99491d0]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260807 03:37:28.465984 16586 tablet_service.cc:1467] Processing CreateTablet for tablet a4e0d750de56427e8a0af5bc030271c4 (DEFAULT_TABLE table=test-workload [id=b66c94eff33b468d8b82a87be99491d0]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260807 03:37:28.467581 16585 tablet_service.cc:1467] Processing CreateTablet for tablet 242a724d826340b693e5b0767a1ba28c (DEFAULT_TABLE table=test-workload [id=b66c94eff33b468d8b82a87be99491d0]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260807 03:37:28.461405 16591 tablet_service.cc:1467] Processing CreateTablet for tablet d07d00a26d54438392d5db566a17627b (DEFAULT_TABLE table=test-workload [id=b66c94eff33b468d8b82a87be99491d0]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260807 03:37:28.470163 16589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10452db9b82745b68c6c353b25eff5ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:28.470866 16587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e409bda14f142959c8127f3bd2643fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:28.476442 16591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d07d00a26d54438392d5db566a17627b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:28.478657 16585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 242a724d826340b693e5b0767a1ba28c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:28.480535 16595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0191ce545e0490986331a9c562b056d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:28.486734 16586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4e0d750de56427e8a0af5bc030271c4. 1 dirs total, 0 dirs full, 0 dirs failed
W20260807 03:37:28.506942 16279 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 03:37:28.519776 16348 tablet_service.cc:1467] Processing CreateTablet for tablet fe184e38bf0d4e51bf42238d161ac738 (DEFAULT_TABLE table=test-workload [id=b66c94eff33b468d8b82a87be99491d0]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260807 03:37:28.532481 16351 tablet_service.cc:1467] Processing CreateTablet for tablet f0191ce545e0490986331a9c562b056d (DEFAULT_TABLE table=test-workload [id=b66c94eff33b468d8b82a87be99491d0]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260807 03:37:28.528460 16348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe184e38bf0d4e51bf42238d161ac738. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:28.542420 16684 tablet_bootstrap.cc:492] T 242a724d826340b693e5b0767a1ba28c P 05aa1d36d28a48138040f6c78c4b34a5: Bootstrap starting.
I20260807 03:37:28.543633 16345 tablet_service.cc:1467] Processing CreateTablet for tablet 485e8d9a337d4e4f9207f737e3a8812b (DEFAULT_TABLE table=test-workload [id=b66c94eff33b468d8b82a87be99491d0]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260807 03:37:28.550902 16351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0191ce545e0490986331a9c562b056d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:28.553022 16344 tablet_service.cc:1467] Processing CreateTablet for tablet d07d00a26d54438392d5db566a17627b (DEFAULT_TABLE table=test-workload [id=b66c94eff33b468d8b82a87be99491d0]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260807 03:37:28.561394 16335 tablet_service.cc:1467] Processing CreateTablet for tablet 242a724d826340b693e5b0767a1ba28c (DEFAULT_TABLE table=test-workload [id=b66c94eff33b468d8b82a87be99491d0]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260807 03:37:28.553694 16341 tablet_service.cc:1467] Processing CreateTablet for tablet 10452db9b82745b68c6c353b25eff5ef (DEFAULT_TABLE table=test-workload [id=b66c94eff33b468d8b82a87be99491d0]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260807 03:37:28.556833 16346 tablet_service.cc:1467] Processing CreateTablet for tablet e9e0fd324ec84973b604e22e851d7b49 (DEFAULT_TABLE table=test-workload [id=b66c94eff33b468d8b82a87be99491d0]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260807 03:37:28.561328 16337 tablet_service.cc:1467] Processing CreateTablet for tablet a4e0d750de56427e8a0af5bc030271c4 (DEFAULT_TABLE table=test-workload [id=b66c94eff33b468d8b82a87be99491d0]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260807 03:37:28.566471 16340 tablet_service.cc:1467] Processing CreateTablet for tablet 5e409bda14f142959c8127f3bd2643fc (DEFAULT_TABLE table=test-workload [id=b66c94eff33b468d8b82a87be99491d0]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260807 03:37:28.577379 16345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 485e8d9a337d4e4f9207f737e3a8812b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:28.581360 16344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d07d00a26d54438392d5db566a17627b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:28.582803 16341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10452db9b82745b68c6c353b25eff5ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:28.585796 16335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 242a724d826340b693e5b0767a1ba28c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:28.592906 16346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9e0fd324ec84973b604e22e851d7b49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:28.604007 16337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4e0d750de56427e8a0af5bc030271c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:28.605548 16340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e409bda14f142959c8127f3bd2643fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:28.651971 16686 tablet_bootstrap.cc:492] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b: Bootstrap starting.
I20260807 03:37:28.680069 16684 tablet_bootstrap.cc:654] T 242a724d826340b693e5b0767a1ba28c P 05aa1d36d28a48138040f6c78c4b34a5: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:28.690407 16687 tablet_bootstrap.cc:492] T f0191ce545e0490986331a9c562b056d P 9ca3bab29b4c4dce908de0247c715423: Bootstrap starting.
I20260807 03:37:28.706194 16686 tablet_bootstrap.cc:654] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:28.712198 16687 tablet_bootstrap.cc:654] T f0191ce545e0490986331a9c562b056d P 9ca3bab29b4c4dce908de0247c715423: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:28.713382 16684 tablet_bootstrap.cc:492] T 242a724d826340b693e5b0767a1ba28c P 05aa1d36d28a48138040f6c78c4b34a5: No bootstrap required, opened a new log
I20260807 03:37:28.713989 16684 ts_tablet_manager.cc:1397] T 242a724d826340b693e5b0767a1ba28c P 05aa1d36d28a48138040f6c78c4b34a5: Time spent bootstrapping tablet: real 0.173s user 0.014s sys 0.006s
I20260807 03:37:28.717522 16684 raft_consensus.cc:348] T 242a724d826340b693e5b0767a1ba28c P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:28.719341 16684 raft_consensus.cc:374] T 242a724d826340b693e5b0767a1ba28c P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:28.719656 16684 raft_consensus.cc:729] T 242a724d826340b693e5b0767a1ba28c P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05aa1d36d28a48138040f6c78c4b34a5, State: Initialized, Role: FOLLOWER
I20260807 03:37:28.720397 16684 consensus_queue.cc:260] T 242a724d826340b693e5b0767a1ba28c P 05aa1d36d28a48138040f6c78c4b34a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:28.720813 16686 tablet_bootstrap.cc:492] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b: No bootstrap required, opened a new log
I20260807 03:37:28.721385 16686 ts_tablet_manager.cc:1397] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b: Time spent bootstrapping tablet: real 0.070s user 0.009s sys 0.008s
I20260807 03:37:28.723342 16687 tablet_bootstrap.cc:492] T f0191ce545e0490986331a9c562b056d P 9ca3bab29b4c4dce908de0247c715423: No bootstrap required, opened a new log
I20260807 03:37:28.723754 16687 ts_tablet_manager.cc:1397] T f0191ce545e0490986331a9c562b056d P 9ca3bab29b4c4dce908de0247c715423: Time spent bootstrapping tablet: real 0.034s user 0.007s sys 0.008s
I20260807 03:37:28.725584 16686 raft_consensus.cc:348] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:28.726572 16686 raft_consensus.cc:374] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:28.726879 16686 raft_consensus.cc:729] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec8c86cff51f48bf9ecc5177e6266b7b, State: Initialized, Role: FOLLOWER
I20260807 03:37:28.727046 16684 ts_tablet_manager.cc:1428] T 242a724d826340b693e5b0767a1ba28c P 05aa1d36d28a48138040f6c78c4b34a5: Time spent starting tablet: real 0.013s user 0.007s sys 0.004s
I20260807 03:37:28.727557 16686 consensus_queue.cc:260] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:28.727875 16687 raft_consensus.cc:348] T f0191ce545e0490986331a9c562b056d P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:28.728674 16687 raft_consensus.cc:374] T f0191ce545e0490986331a9c562b056d P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:28.728909 16526 heartbeater.cc:503] Master 127.15.198.190:41955 was elected leader, sending a full tablet report...
I20260807 03:37:28.728968 16687 raft_consensus.cc:729] T f0191ce545e0490986331a9c562b056d P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ca3bab29b4c4dce908de0247c715423, State: Initialized, Role: FOLLOWER
I20260807 03:37:28.729601 16687 consensus_queue.cc:260] T f0191ce545e0490986331a9c562b056d P 9ca3bab29b4c4dce908de0247c715423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:28.733484 16409 heartbeater.cc:503] Master 127.15.198.190:41955 was elected leader, sending a full tablet report...
I20260807 03:37:28.734892 16687 ts_tablet_manager.cc:1428] T f0191ce545e0490986331a9c562b056d P 9ca3bab29b4c4dce908de0247c715423: Time spent starting tablet: real 0.011s user 0.000s sys 0.010s
I20260807 03:37:28.735906 16687 tablet_bootstrap.cc:492] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423: Bootstrap starting.
I20260807 03:37:28.736524 16684 tablet_bootstrap.cc:492] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5: Bootstrap starting.
I20260807 03:37:28.742239 16687 tablet_bootstrap.cc:654] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:28.754258 16687 tablet_bootstrap.cc:492] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423: No bootstrap required, opened a new log
I20260807 03:37:28.754734 16687 ts_tablet_manager.cc:1397] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.005s
I20260807 03:37:28.757515 16687 raft_consensus.cc:348] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:28.758074 16687 raft_consensus.cc:374] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:28.758332 16687 raft_consensus.cc:729] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ca3bab29b4c4dce908de0247c715423, State: Initialized, Role: FOLLOWER
I20260807 03:37:28.758930 16687 consensus_queue.cc:260] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:28.760730 16687 ts_tablet_manager.cc:1428] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260807 03:37:28.761628 16687 tablet_bootstrap.cc:492] T 485e8d9a337d4e4f9207f737e3a8812b P 9ca3bab29b4c4dce908de0247c715423: Bootstrap starting.
I20260807 03:37:28.762953 16686 ts_tablet_manager.cc:1428] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b: Time spent starting tablet: real 0.041s user 0.008s sys 0.002s
I20260807 03:37:28.767457 16687 tablet_bootstrap.cc:654] T 485e8d9a337d4e4f9207f737e3a8812b P 9ca3bab29b4c4dce908de0247c715423: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:28.770346 16684 tablet_bootstrap.cc:654] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:28.773274 16646 heartbeater.cc:503] Master 127.15.198.190:41955 was elected leader, sending a full tablet report...
I20260807 03:37:28.777393 16686 tablet_bootstrap.cc:492] T 485e8d9a337d4e4f9207f737e3a8812b P ec8c86cff51f48bf9ecc5177e6266b7b: Bootstrap starting.
I20260807 03:37:28.779340 16687 tablet_bootstrap.cc:492] T 485e8d9a337d4e4f9207f737e3a8812b P 9ca3bab29b4c4dce908de0247c715423: No bootstrap required, opened a new log
I20260807 03:37:28.779850 16687 ts_tablet_manager.cc:1397] T 485e8d9a337d4e4f9207f737e3a8812b P 9ca3bab29b4c4dce908de0247c715423: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.008s
I20260807 03:37:28.786058 16684 tablet_bootstrap.cc:492] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5: No bootstrap required, opened a new log
I20260807 03:37:28.785972 16687 raft_consensus.cc:348] T 485e8d9a337d4e4f9207f737e3a8812b P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:28.786993 16684 ts_tablet_manager.cc:1397] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.009s
I20260807 03:37:28.786940 16687 raft_consensus.cc:374] T 485e8d9a337d4e4f9207f737e3a8812b P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:28.787925 16687 raft_consensus.cc:729] T 485e8d9a337d4e4f9207f737e3a8812b P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ca3bab29b4c4dce908de0247c715423, State: Initialized, Role: FOLLOWER
I20260807 03:37:28.789312 16687 consensus_queue.cc:260] T 485e8d9a337d4e4f9207f737e3a8812b P 9ca3bab29b4c4dce908de0247c715423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:28.792615 16684 raft_consensus.cc:348] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:28.793423 16684 raft_consensus.cc:374] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:28.791719 16686 tablet_bootstrap.cc:654] T 485e8d9a337d4e4f9207f737e3a8812b P ec8c86cff51f48bf9ecc5177e6266b7b: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:28.793711 16684 raft_consensus.cc:729] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05aa1d36d28a48138040f6c78c4b34a5, State: Initialized, Role: FOLLOWER
I20260807 03:37:28.794431 16687 ts_tablet_manager.cc:1428] T 485e8d9a337d4e4f9207f737e3a8812b P 9ca3bab29b4c4dce908de0247c715423: Time spent starting tablet: real 0.014s user 0.008s sys 0.000s
I20260807 03:37:28.794497 16684 consensus_queue.cc:260] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:28.797900 16687 tablet_bootstrap.cc:492] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423: Bootstrap starting.
W20260807 03:37:28.798174 16528 tablet.cc:2369] T 242a724d826340b693e5b0767a1ba28c P 05aa1d36d28a48138040f6c78c4b34a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:37:28.801345 16684 ts_tablet_manager.cc:1428] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5: Time spent starting tablet: real 0.014s user 0.005s sys 0.004s
I20260807 03:37:28.805807 16684 tablet_bootstrap.cc:492] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5: Bootstrap starting.
I20260807 03:37:28.816004 16686 tablet_bootstrap.cc:492] T 485e8d9a337d4e4f9207f737e3a8812b P ec8c86cff51f48bf9ecc5177e6266b7b: No bootstrap required, opened a new log
I20260807 03:37:28.832295 16686 ts_tablet_manager.cc:1397] T 485e8d9a337d4e4f9207f737e3a8812b P ec8c86cff51f48bf9ecc5177e6266b7b: Time spent bootstrapping tablet: real 0.055s user 0.007s sys 0.014s
I20260807 03:37:28.834052 16687 tablet_bootstrap.cc:654] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:28.834519 16684 tablet_bootstrap.cc:654] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:28.840518 16686 raft_consensus.cc:348] T 485e8d9a337d4e4f9207f737e3a8812b P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:28.841179 16686 raft_consensus.cc:374] T 485e8d9a337d4e4f9207f737e3a8812b P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:28.841375 16686 raft_consensus.cc:729] T 485e8d9a337d4e4f9207f737e3a8812b P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec8c86cff51f48bf9ecc5177e6266b7b, State: Initialized, Role: FOLLOWER
I20260807 03:37:28.841897 16686 consensus_queue.cc:260] T 485e8d9a337d4e4f9207f737e3a8812b P ec8c86cff51f48bf9ecc5177e6266b7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:28.842142 16687 tablet_bootstrap.cc:492] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423: No bootstrap required, opened a new log
I20260807 03:37:28.842641 16687 ts_tablet_manager.cc:1397] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423: Time spent bootstrapping tablet: real 0.046s user 0.021s sys 0.011s
I20260807 03:37:28.843780 16686 ts_tablet_manager.cc:1428] T 485e8d9a337d4e4f9207f737e3a8812b P ec8c86cff51f48bf9ecc5177e6266b7b: Time spent starting tablet: real 0.011s user 0.005s sys 0.001s
I20260807 03:37:28.845939 16695 raft_consensus.cc:482] T 485e8d9a337d4e4f9207f737e3a8812b P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:37:28.846259 16684 tablet_bootstrap.cc:492] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5: No bootstrap required, opened a new log
I20260807 03:37:28.846794 16684 ts_tablet_manager.cc:1397] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5: Time spent bootstrapping tablet: real 0.041s user 0.012s sys 0.001s
I20260807 03:37:28.846542 16695 raft_consensus.cc:504] T 485e8d9a337d4e4f9207f737e3a8812b P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:28.849665 16695 leader_election.cc:288] T 485e8d9a337d4e4f9207f737e3a8812b P 9ca3bab29b4c4dce908de0247c715423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130:39359), ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131:37775)
I20260807 03:37:28.849475 16684 raft_consensus.cc:348] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:28.850210 16684 raft_consensus.cc:374] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:28.850564 16684 raft_consensus.cc:729] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05aa1d36d28a48138040f6c78c4b34a5, State: Initialized, Role: FOLLOWER
I20260807 03:37:28.851326 16684 consensus_queue.cc:260] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:28.853932 16684 ts_tablet_manager.cc:1428] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20260807 03:37:28.855122 16686 tablet_bootstrap.cc:492] T 5e409bda14f142959c8127f3bd2643fc P ec8c86cff51f48bf9ecc5177e6266b7b: Bootstrap starting.
I20260807 03:37:28.854775 16684 tablet_bootstrap.cc:492] T a4e0d750de56427e8a0af5bc030271c4 P 05aa1d36d28a48138040f6c78c4b34a5: Bootstrap starting.
I20260807 03:37:28.864022 16684 tablet_bootstrap.cc:654] T a4e0d750de56427e8a0af5bc030271c4 P 05aa1d36d28a48138040f6c78c4b34a5: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:28.845918 16687 raft_consensus.cc:348] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:28.865316 16687 raft_consensus.cc:374] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:28.865644 16687 raft_consensus.cc:729] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ca3bab29b4c4dce908de0247c715423, State: Initialized, Role: FOLLOWER
I20260807 03:37:28.866405 16687 consensus_queue.cc:260] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:28.869297 16686 tablet_bootstrap.cc:654] T 5e409bda14f142959c8127f3bd2643fc P ec8c86cff51f48bf9ecc5177e6266b7b: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:28.872344 16687 ts_tablet_manager.cc:1428] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423: Time spent starting tablet: real 0.029s user 0.007s sys 0.001s
I20260807 03:37:28.873312 16687 tablet_bootstrap.cc:492] T 5e409bda14f142959c8127f3bd2643fc P 9ca3bab29b4c4dce908de0247c715423: Bootstrap starting.
I20260807 03:37:28.892843 16686 tablet_bootstrap.cc:492] T 5e409bda14f142959c8127f3bd2643fc P ec8c86cff51f48bf9ecc5177e6266b7b: No bootstrap required, opened a new log
I20260807 03:37:28.893298 16686 ts_tablet_manager.cc:1397] T 5e409bda14f142959c8127f3bd2643fc P ec8c86cff51f48bf9ecc5177e6266b7b: Time spent bootstrapping tablet: real 0.038s user 0.007s sys 0.013s
I20260807 03:37:28.896589 16687 tablet_bootstrap.cc:654] T 5e409bda14f142959c8127f3bd2643fc P 9ca3bab29b4c4dce908de0247c715423: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:28.898037 16684 tablet_bootstrap.cc:492] T a4e0d750de56427e8a0af5bc030271c4 P 05aa1d36d28a48138040f6c78c4b34a5: No bootstrap required, opened a new log
I20260807 03:37:28.897461 16686 raft_consensus.cc:348] T 5e409bda14f142959c8127f3bd2643fc P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:28.898936 16695 raft_consensus.cc:482] T f0191ce545e0490986331a9c562b056d P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:37:28.899544 16686 raft_consensus.cc:374] T 5e409bda14f142959c8127f3bd2643fc P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:28.899737 16684 ts_tablet_manager.cc:1397] T a4e0d750de56427e8a0af5bc030271c4 P 05aa1d36d28a48138040f6c78c4b34a5: Time spent bootstrapping tablet: real 0.045s user 0.013s sys 0.019s
I20260807 03:37:28.899907 16686 raft_consensus.cc:729] T 5e409bda14f142959c8127f3bd2643fc P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec8c86cff51f48bf9ecc5177e6266b7b, State: Initialized, Role: FOLLOWER
I20260807 03:37:28.899698 16695 raft_consensus.cc:504] T f0191ce545e0490986331a9c562b056d P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:28.900744 16686 consensus_queue.cc:260] T 5e409bda14f142959c8127f3bd2643fc P ec8c86cff51f48bf9ecc5177e6266b7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:28.903059 16695 leader_election.cc:288] T f0191ce545e0490986331a9c562b056d P 9ca3bab29b4c4dce908de0247c715423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130:39359), ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131:37775)
I20260807 03:37:28.904440 16684 raft_consensus.cc:348] T a4e0d750de56427e8a0af5bc030271c4 P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:28.905143 16684 raft_consensus.cc:374] T a4e0d750de56427e8a0af5bc030271c4 P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:28.905463 16684 raft_consensus.cc:729] T a4e0d750de56427e8a0af5bc030271c4 P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05aa1d36d28a48138040f6c78c4b34a5, State: Initialized, Role: FOLLOWER
I20260807 03:37:28.906113 16684 consensus_queue.cc:260] T a4e0d750de56427e8a0af5bc030271c4 P 05aa1d36d28a48138040f6c78c4b34a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:28.910233 16694 raft_consensus.cc:482] T a4e0d750de56427e8a0af5bc030271c4 P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:37:28.910737 16694 raft_consensus.cc:504] T a4e0d750de56427e8a0af5bc030271c4 P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:28.913369 16694 leader_election.cc:288] T a4e0d750de56427e8a0af5bc030271c4 P 05aa1d36d28a48138040f6c78c4b34a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129:36375), ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131:37775)
I20260807 03:37:28.915844 16686 ts_tablet_manager.cc:1428] T 5e409bda14f142959c8127f3bd2643fc P ec8c86cff51f48bf9ecc5177e6266b7b: Time spent starting tablet: real 0.022s user 0.005s sys 0.004s
I20260807 03:37:28.916831 16684 ts_tablet_manager.cc:1428] T a4e0d750de56427e8a0af5bc030271c4 P 05aa1d36d28a48138040f6c78c4b34a5: Time spent starting tablet: real 0.017s user 0.005s sys 0.002s
I20260807 03:37:28.918572 16696 raft_consensus.cc:482] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:37:28.919023 16696 raft_consensus.cc:504] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:28.924746 16696 leader_election.cc:288] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129:36375), 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130:39359)
I20260807 03:37:28.925222 16694 raft_consensus.cc:482] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:37:28.927484 16686 tablet_bootstrap.cc:492] T a4e0d750de56427e8a0af5bc030271c4 P ec8c86cff51f48bf9ecc5177e6266b7b: Bootstrap starting.
I20260807 03:37:28.927138 16684 tablet_bootstrap.cc:492] T d07d00a26d54438392d5db566a17627b P 05aa1d36d28a48138040f6c78c4b34a5: Bootstrap starting.
I20260807 03:37:28.929744 16481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0191ce545e0490986331a9c562b056d" candidate_uuid: "9ca3bab29b4c4dce908de0247c715423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05aa1d36d28a48138040f6c78c4b34a5" is_pre_election: true
I20260807 03:37:28.928508 16484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "485e8d9a337d4e4f9207f737e3a8812b" candidate_uuid: "9ca3bab29b4c4dce908de0247c715423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05aa1d36d28a48138040f6c78c4b34a5" is_pre_election: true
I20260807 03:37:28.926510 16694 raft_consensus.cc:504] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
W20260807 03:37:28.934497 16314 leader_election.cc:341] T 485e8d9a337d4e4f9207f737e3a8812b P 9ca3bab29b4c4dce908de0247c715423 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:37:28.934695 16694 leader_election.cc:288] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129:36375), ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131:37775)
W20260807 03:37:28.935247 16314 leader_election.cc:341] T f0191ce545e0490986331a9c562b056d P 9ca3bab29b4c4dce908de0247c715423 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:37:28.941107 16684 tablet_bootstrap.cc:654] T d07d00a26d54438392d5db566a17627b P 05aa1d36d28a48138040f6c78c4b34a5: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:28.961390 16686 tablet_bootstrap.cc:654] T a4e0d750de56427e8a0af5bc030271c4 P ec8c86cff51f48bf9ecc5177e6266b7b: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:28.976195 16607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "485e8d9a337d4e4f9207f737e3a8812b" candidate_uuid: "9ca3bab29b4c4dce908de0247c715423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" is_pre_election: true
I20260807 03:37:28.977178 16607 raft_consensus.cc:2454] T 485e8d9a337d4e4f9207f737e3a8812b P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ca3bab29b4c4dce908de0247c715423 in term 0.
I20260807 03:37:28.978881 16607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0191ce545e0490986331a9c562b056d" candidate_uuid: "9ca3bab29b4c4dce908de0247c715423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" is_pre_election: true
I20260807 03:37:28.980085 16684 tablet_bootstrap.cc:492] T d07d00a26d54438392d5db566a17627b P 05aa1d36d28a48138040f6c78c4b34a5: No bootstrap required, opened a new log
I20260807 03:37:28.980669 16684 ts_tablet_manager.cc:1397] T d07d00a26d54438392d5db566a17627b P 05aa1d36d28a48138040f6c78c4b34a5: Time spent bootstrapping tablet: real 0.055s user 0.007s sys 0.013s
I20260807 03:37:28.980978 16314 leader_election.cc:302] T 485e8d9a337d4e4f9207f737e3a8812b P 9ca3bab29b4c4dce908de0247c715423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ca3bab29b4c4dce908de0247c715423, ec8c86cff51f48bf9ecc5177e6266b7b; no voters: 05aa1d36d28a48138040f6c78c4b34a5
I20260807 03:37:28.981863 16695 raft_consensus.cc:2781] T 485e8d9a337d4e4f9207f737e3a8812b P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:37:28.982203 16695 raft_consensus.cc:482] T 485e8d9a337d4e4f9207f737e3a8812b P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:37:28.982554 16695 raft_consensus.cc:3037] T 485e8d9a337d4e4f9207f737e3a8812b P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Advancing to term 1
W20260807 03:37:28.982648 16314 leader_election.cc:341] T f0191ce545e0490986331a9c562b056d P 9ca3bab29b4c4dce908de0247c715423 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131:37775): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:37:28.983058 16314 leader_election.cc:302] T f0191ce545e0490986331a9c562b056d P 9ca3bab29b4c4dce908de0247c715423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ca3bab29b4c4dce908de0247c715423; no voters: 05aa1d36d28a48138040f6c78c4b34a5, ec8c86cff51f48bf9ecc5177e6266b7b
I20260807 03:37:28.983961 16684 raft_consensus.cc:348] T d07d00a26d54438392d5db566a17627b P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:28.984880 16684 raft_consensus.cc:374] T d07d00a26d54438392d5db566a17627b P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:28.985255 16684 raft_consensus.cc:729] T d07d00a26d54438392d5db566a17627b P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05aa1d36d28a48138040f6c78c4b34a5, State: Initialized, Role: FOLLOWER
I20260807 03:37:28.986097 16684 consensus_queue.cc:260] T d07d00a26d54438392d5db566a17627b P 05aa1d36d28a48138040f6c78c4b34a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:28.991115 16695 raft_consensus.cc:504] T 485e8d9a337d4e4f9207f737e3a8812b P 9ca3bab29b4c4dce908de0247c715423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.002775 16686 tablet_bootstrap.cc:492] T a4e0d750de56427e8a0af5bc030271c4 P ec8c86cff51f48bf9ecc5177e6266b7b: No bootstrap required, opened a new log
I20260807 03:37:29.003459 16686 ts_tablet_manager.cc:1397] T a4e0d750de56427e8a0af5bc030271c4 P ec8c86cff51f48bf9ecc5177e6266b7b: Time spent bootstrapping tablet: real 0.076s user 0.010s sys 0.010s
I20260807 03:37:29.004812 16687 tablet_bootstrap.cc:492] T 5e409bda14f142959c8127f3bd2643fc P 9ca3bab29b4c4dce908de0247c715423: No bootstrap required, opened a new log
I20260807 03:37:29.005455 16687 ts_tablet_manager.cc:1397] T 5e409bda14f142959c8127f3bd2643fc P 9ca3bab29b4c4dce908de0247c715423: Time spent bootstrapping tablet: real 0.132s user 0.002s sys 0.024s
I20260807 03:37:29.010519 16695 leader_election.cc:288] T 485e8d9a337d4e4f9207f737e3a8812b P 9ca3bab29b4c4dce908de0247c715423 [CANDIDATE]: Term 1 election: Requested vote from peers 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130:39359), ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131:37775)
I20260807 03:37:29.011205 16695 raft_consensus.cc:2726] T f0191ce545e0490986331a9c562b056d P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:37:29.011163 16481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "485e8d9a337d4e4f9207f737e3a8812b" candidate_uuid: "9ca3bab29b4c4dce908de0247c715423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05aa1d36d28a48138040f6c78c4b34a5"
I20260807 03:37:29.011402 16687 raft_consensus.cc:348] T 5e409bda14f142959c8127f3bd2643fc P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.013556 16607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "485e8d9a337d4e4f9207f737e3a8812b" candidate_uuid: "9ca3bab29b4c4dce908de0247c715423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b"
I20260807 03:37:29.013525 16687 raft_consensus.cc:374] T 5e409bda14f142959c8127f3bd2643fc P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:29.014835 16684 ts_tablet_manager.cc:1428] T d07d00a26d54438392d5db566a17627b P 05aa1d36d28a48138040f6c78c4b34a5: Time spent starting tablet: real 0.034s user 0.009s sys 0.000s
I20260807 03:37:29.014912 16607 raft_consensus.cc:3037] T 485e8d9a337d4e4f9207f737e3a8812b P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:37:29.014930 16687 raft_consensus.cc:729] T 5e409bda14f142959c8127f3bd2643fc P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ca3bab29b4c4dce908de0247c715423, State: Initialized, Role: FOLLOWER
I20260807 03:37:29.016639 16684 tablet_bootstrap.cc:492] T fe184e38bf0d4e51bf42238d161ac738 P 05aa1d36d28a48138040f6c78c4b34a5: Bootstrap starting.
W20260807 03:37:29.015686 16314 leader_election.cc:341] T 485e8d9a337d4e4f9207f737e3a8812b P 9ca3bab29b4c4dce908de0247c715423 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:37:29.023576 16686 raft_consensus.cc:348] T a4e0d750de56427e8a0af5bc030271c4 P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.024336 16686 raft_consensus.cc:374] T a4e0d750de56427e8a0af5bc030271c4 P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:29.024624 16686 raft_consensus.cc:729] T a4e0d750de56427e8a0af5bc030271c4 P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec8c86cff51f48bf9ecc5177e6266b7b, State: Initialized, Role: FOLLOWER
I20260807 03:37:29.025312 16686 consensus_queue.cc:260] T a4e0d750de56427e8a0af5bc030271c4 P ec8c86cff51f48bf9ecc5177e6266b7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.021423 16687 consensus_queue.cc:260] T 5e409bda14f142959c8127f3bd2643fc P 9ca3bab29b4c4dce908de0247c715423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.026721 16684 tablet_bootstrap.cc:654] T fe184e38bf0d4e51bf42238d161ac738 P 05aa1d36d28a48138040f6c78c4b34a5: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:29.030709 16607 raft_consensus.cc:2454] T 485e8d9a337d4e4f9207f737e3a8812b P ec8c86cff51f48bf9ecc5177e6266b7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ca3bab29b4c4dce908de0247c715423 in term 1.
I20260807 03:37:29.032065 16314 leader_election.cc:302] T 485e8d9a337d4e4f9207f737e3a8812b P 9ca3bab29b4c4dce908de0247c715423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ca3bab29b4c4dce908de0247c715423, ec8c86cff51f48bf9ecc5177e6266b7b; no voters: 05aa1d36d28a48138040f6c78c4b34a5
I20260807 03:37:29.033995 16695 raft_consensus.cc:2781] T 485e8d9a337d4e4f9207f737e3a8812b P 9ca3bab29b4c4dce908de0247c715423 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:37:29.033634 16687 ts_tablet_manager.cc:1428] T 5e409bda14f142959c8127f3bd2643fc P 9ca3bab29b4c4dce908de0247c715423: Time spent starting tablet: real 0.028s user 0.008s sys 0.000s
I20260807 03:37:29.036378 16687 tablet_bootstrap.cc:492] T fe184e38bf0d4e51bf42238d161ac738 P 9ca3bab29b4c4dce908de0247c715423: Bootstrap starting.
I20260807 03:37:29.036227 16729 raft_consensus.cc:482] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:37:29.036787 16729 raft_consensus.cc:504] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.039050 16729 leader_election.cc:288] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130:39359), ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131:37775)
I20260807 03:37:29.046738 16729 raft_consensus.cc:482] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:37:29.046757 16481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10452db9b82745b68c6c353b25eff5ef" candidate_uuid: "9ca3bab29b4c4dce908de0247c715423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05aa1d36d28a48138040f6c78c4b34a5" is_pre_election: true
I20260807 03:37:29.049032 16481 raft_consensus.cc:2454] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ca3bab29b4c4dce908de0247c715423 in term 0.
I20260807 03:37:29.045596 16607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10452db9b82745b68c6c353b25eff5ef" candidate_uuid: "9ca3bab29b4c4dce908de0247c715423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" is_pre_election: true
I20260807 03:37:29.052675 16314 leader_election.cc:302] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05aa1d36d28a48138040f6c78c4b34a5, 9ca3bab29b4c4dce908de0247c715423; no voters:
I20260807 03:37:29.048478 16729 raft_consensus.cc:504] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.062441 16694 raft_consensus.cc:482] T 242a724d826340b693e5b0767a1ba28c P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 03:37:29.064085 16314 leader_election.cc:341] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131:37775): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:37:29.063655 16694 raft_consensus.cc:504] T 242a724d826340b693e5b0767a1ba28c P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.067570 16733 raft_consensus.cc:2781] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:37:29.063613 16695 raft_consensus.cc:686] T 485e8d9a337d4e4f9207f737e3a8812b P 9ca3bab29b4c4dce908de0247c715423 [term 1 LEADER]: Becoming Leader. State: Replica: 9ca3bab29b4c4dce908de0247c715423, State: Running, Role: LEADER
I20260807 03:37:29.069969 16686 ts_tablet_manager.cc:1428] T a4e0d750de56427e8a0af5bc030271c4 P ec8c86cff51f48bf9ecc5177e6266b7b: Time spent starting tablet: real 0.066s user 0.007s sys 0.002s
I20260807 03:37:29.068032 16733 raft_consensus.cc:482] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:37:29.068964 16694 leader_election.cc:288] T 242a724d826340b693e5b0767a1ba28c P 05aa1d36d28a48138040f6c78c4b34a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129:36375), ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131:37775)
I20260807 03:37:29.070693 16695 consensus_queue.cc:237] T 485e8d9a337d4e4f9207f737e3a8812b P 9ca3bab29b4c4dce908de0247c715423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.073657 16733 raft_consensus.cc:3037] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:37:29.079820 16686 tablet_bootstrap.cc:492] T d07d00a26d54438392d5db566a17627b P ec8c86cff51f48bf9ecc5177e6266b7b: Bootstrap starting.
I20260807 03:37:29.077754 16367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4e0d750de56427e8a0af5bc030271c4" candidate_uuid: "05aa1d36d28a48138040f6c78c4b34a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ca3bab29b4c4dce908de0247c715423" is_pre_election: true
I20260807 03:37:29.078068 16481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e0fd324ec84973b604e22e851d7b49" candidate_uuid: "9ca3bab29b4c4dce908de0247c715423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05aa1d36d28a48138040f6c78c4b34a5" is_pre_election: true
I20260807 03:37:29.079294 16729 leader_election.cc:288] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130:39359), ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131:37775)
I20260807 03:37:29.079622 16684 tablet_bootstrap.cc:492] T fe184e38bf0d4e51bf42238d161ac738 P 05aa1d36d28a48138040f6c78c4b34a5: No bootstrap required, opened a new log
I20260807 03:37:29.079977 16687 tablet_bootstrap.cc:654] T fe184e38bf0d4e51bf42238d161ac738 P 9ca3bab29b4c4dce908de0247c715423: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:29.084586 16481 raft_consensus.cc:2454] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ca3bab29b4c4dce908de0247c715423 in term 0.
I20260807 03:37:29.086755 16684 ts_tablet_manager.cc:1397] T fe184e38bf0d4e51bf42238d161ac738 P 05aa1d36d28a48138040f6c78c4b34a5: Time spent bootstrapping tablet: real 0.071s user 0.022s sys 0.022s
I20260807 03:37:29.085566 16607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e0fd324ec84973b604e22e851d7b49" candidate_uuid: "9ca3bab29b4c4dce908de0247c715423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" is_pre_election: true
I20260807 03:37:29.087495 16365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e0fd324ec84973b604e22e851d7b49" candidate_uuid: "05aa1d36d28a48138040f6c78c4b34a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ca3bab29b4c4dce908de0247c715423" is_pre_election: true
I20260807 03:37:29.089298 16364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242a724d826340b693e5b0767a1ba28c" candidate_uuid: "05aa1d36d28a48138040f6c78c4b34a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ca3bab29b4c4dce908de0247c715423" is_pre_election: true
I20260807 03:37:29.090528 16365 raft_consensus.cc:2454] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05aa1d36d28a48138040f6c78c4b34a5 in term 0.
W20260807 03:37:29.094010 16432 leader_election.cc:341] T 242a724d826340b693e5b0767a1ba28c P 05aa1d36d28a48138040f6c78c4b34a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129:36375): Illegal state: must be running to vote when last-logged opid is not known
W20260807 03:37:29.094736 16432 leader_election.cc:341] T a4e0d750de56427e8a0af5bc030271c4 P 05aa1d36d28a48138040f6c78c4b34a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129:36375): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:37:29.088929 16314 leader_election.cc:302] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05aa1d36d28a48138040f6c78c4b34a5, 9ca3bab29b4c4dce908de0247c715423; no voters:
W20260807 03:37:29.096494 16314 leader_election.cc:341] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131:37775): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:37:29.097399 16432 leader_election.cc:302] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05aa1d36d28a48138040f6c78c4b34a5, 9ca3bab29b4c4dce908de0247c715423; no voters:
I20260807 03:37:29.097301 16684 raft_consensus.cc:348] T fe184e38bf0d4e51bf42238d161ac738 P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.099334 16684 raft_consensus.cc:374] T fe184e38bf0d4e51bf42238d161ac738 P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:29.099695 16684 raft_consensus.cc:729] T fe184e38bf0d4e51bf42238d161ac738 P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05aa1d36d28a48138040f6c78c4b34a5, State: Initialized, Role: FOLLOWER
I20260807 03:37:29.099831 16694 raft_consensus.cc:2781] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:37:29.103395 16694 raft_consensus.cc:482] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:37:29.098891 16729 raft_consensus.cc:2781] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:37:29.108405 16729 raft_consensus.cc:482] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:37:29.108740 16694 raft_consensus.cc:3037] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:37:29.101962 16684 consensus_queue.cc:260] T fe184e38bf0d4e51bf42238d161ac738 P 05aa1d36d28a48138040f6c78c4b34a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.111217 16729 raft_consensus.cc:3037] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:37:29.110599 16686 tablet_bootstrap.cc:654] T d07d00a26d54438392d5db566a17627b P ec8c86cff51f48bf9ecc5177e6266b7b: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:29.122916 16481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe184e38bf0d4e51bf42238d161ac738" candidate_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05aa1d36d28a48138040f6c78c4b34a5" is_pre_election: true
I20260807 03:37:29.123555 16481 raft_consensus.cc:2454] T fe184e38bf0d4e51bf42238d161ac738 P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec8c86cff51f48bf9ecc5177e6266b7b in term 0.
I20260807 03:37:29.126698 16557 leader_election.cc:302] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05aa1d36d28a48138040f6c78c4b34a5, ec8c86cff51f48bf9ecc5177e6266b7b; no voters:
I20260807 03:37:29.129081 16365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe184e38bf0d4e51bf42238d161ac738" candidate_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ca3bab29b4c4dce908de0247c715423" is_pre_election: true
I20260807 03:37:29.131551 16696 raft_consensus.cc:2781] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:37:29.132040 16696 raft_consensus.cc:482] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:37:29.132679 16696 raft_consensus.cc:3037] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:37:29.133354 16729 raft_consensus.cc:504] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
W20260807 03:37:29.134481 16556 leader_election.cc:341] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129:36375): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:37:29.137326 16481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e0fd324ec84973b604e22e851d7b49" candidate_uuid: "9ca3bab29b4c4dce908de0247c715423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05aa1d36d28a48138040f6c78c4b34a5"
I20260807 03:37:29.139846 16607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e0fd324ec84973b604e22e851d7b49" candidate_uuid: "9ca3bab29b4c4dce908de0247c715423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b"
W20260807 03:37:29.141471 16314 leader_election.cc:341] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131:37775): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:37:29.142117 16729 leader_election.cc:288] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [CANDIDATE]: Term 1 election: Requested vote from peers 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130:39359), ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131:37775)
I20260807 03:37:29.148948 16604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242a724d826340b693e5b0767a1ba28c" candidate_uuid: "05aa1d36d28a48138040f6c78c4b34a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" is_pre_election: true
I20260807 03:37:29.148679 16606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e0fd324ec84973b604e22e851d7b49" candidate_uuid: "05aa1d36d28a48138040f6c78c4b34a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" is_pre_election: true
W20260807 03:37:29.159540 16433 leader_election.cc:341] T 242a724d826340b693e5b0767a1ba28c P 05aa1d36d28a48138040f6c78c4b34a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131:37775): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:37:29.160167 16433 leader_election.cc:302] T 242a724d826340b693e5b0767a1ba28c P 05aa1d36d28a48138040f6c78c4b34a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 05aa1d36d28a48138040f6c78c4b34a5; no voters: 9ca3bab29b4c4dce908de0247c715423, ec8c86cff51f48bf9ecc5177e6266b7b
I20260807 03:37:29.159173 16607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4e0d750de56427e8a0af5bc030271c4" candidate_uuid: "05aa1d36d28a48138040f6c78c4b34a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" is_pre_election: true
I20260807 03:37:29.163807 16607 raft_consensus.cc:2454] T a4e0d750de56427e8a0af5bc030271c4 P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05aa1d36d28a48138040f6c78c4b34a5 in term 0.
I20260807 03:37:29.152611 16206 catalog_manager.cc:5495] T 485e8d9a337d4e4f9207f737e3a8812b P 9ca3bab29b4c4dce908de0247c715423 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129). New cstate: current_term: 1 leader_uuid: "9ca3bab29b4c4dce908de0247c715423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } health_report { overall_health: UNKNOWN } } }
I20260807 03:37:29.159111 16696 raft_consensus.cc:504] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.165918 16733 raft_consensus.cc:504] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.186206 16733 leader_election.cc:288] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [CANDIDATE]: Term 1 election: Requested vote from peers 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130:39359), ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131:37775)
I20260807 03:37:29.186520 16696 leader_election.cc:288] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b [CANDIDATE]: Term 1 election: Requested vote from peers 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129:36375), 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130:39359)
I20260807 03:37:29.200405 16696 raft_consensus.cc:482] T 5e409bda14f142959c8127f3bd2643fc P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:37:29.201056 16694 raft_consensus.cc:504] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.204197 16696 raft_consensus.cc:504] T 5e409bda14f142959c8127f3bd2643fc P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.204339 16479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe184e38bf0d4e51bf42238d161ac738" candidate_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05aa1d36d28a48138040f6c78c4b34a5"
I20260807 03:37:29.206190 16479 raft_consensus.cc:3037] T fe184e38bf0d4e51bf42238d161ac738 P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:37:29.200938 16607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10452db9b82745b68c6c353b25eff5ef" candidate_uuid: "9ca3bab29b4c4dce908de0247c715423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b"
I20260807 03:37:29.212270 16484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10452db9b82745b68c6c353b25eff5ef" candidate_uuid: "9ca3bab29b4c4dce908de0247c715423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05aa1d36d28a48138040f6c78c4b34a5"
I20260807 03:37:29.259559 16365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e0fd324ec84973b604e22e851d7b49" candidate_uuid: "05aa1d36d28a48138040f6c78c4b34a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ca3bab29b4c4dce908de0247c715423"
I20260807 03:37:29.260344 16365 raft_consensus.cc:2379] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 05aa1d36d28a48138040f6c78c4b34a5 in current term 1: Already voted for candidate 9ca3bab29b4c4dce908de0247c715423 in this term.
I20260807 03:37:29.257392 16481 raft_consensus.cc:2379] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9ca3bab29b4c4dce908de0247c715423 in current term 1: Already voted for candidate 05aa1d36d28a48138040f6c78c4b34a5 in this term.
I20260807 03:37:29.212359 16684 ts_tablet_manager.cc:1428] T fe184e38bf0d4e51bf42238d161ac738 P 05aa1d36d28a48138040f6c78c4b34a5: Time spent starting tablet: real 0.125s user 0.014s sys 0.012s
I20260807 03:37:29.261215 16484 raft_consensus.cc:3037] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:37:29.212450 16696 leader_election.cc:288] T 5e409bda14f142959c8127f3bd2643fc P ec8c86cff51f48bf9ecc5177e6266b7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129:36375), 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130:39359)
I20260807 03:37:29.266311 16477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e409bda14f142959c8127f3bd2643fc" candidate_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05aa1d36d28a48138040f6c78c4b34a5" is_pre_election: true
W20260807 03:37:29.268138 16314 leader_election.cc:341] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131:37775): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:37:29.269054 16684 tablet_bootstrap.cc:492] T 5e409bda14f142959c8127f3bd2643fc P 05aa1d36d28a48138040f6c78c4b34a5: Bootstrap starting.
W20260807 03:37:29.270650 16557 leader_election.cc:341] T 5e409bda14f142959c8127f3bd2643fc P ec8c86cff51f48bf9ecc5177e6266b7b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:37:29.273016 16687 tablet_bootstrap.cc:492] T fe184e38bf0d4e51bf42238d161ac738 P 9ca3bab29b4c4dce908de0247c715423: No bootstrap required, opened a new log
I20260807 03:37:29.273056 16365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe184e38bf0d4e51bf42238d161ac738" candidate_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ca3bab29b4c4dce908de0247c715423"
I20260807 03:37:29.273579 16687 ts_tablet_manager.cc:1397] T fe184e38bf0d4e51bf42238d161ac738 P 9ca3bab29b4c4dce908de0247c715423: Time spent bootstrapping tablet: real 0.237s user 0.011s sys 0.012s
I20260807 03:37:29.265883 16694 leader_election.cc:288] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [CANDIDATE]: Term 1 election: Requested vote from peers 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129:36375), ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131:37775)
I20260807 03:37:29.276859 16687 raft_consensus.cc:348] T fe184e38bf0d4e51bf42238d161ac738 P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.277984 16314 leader_election.cc:302] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ca3bab29b4c4dce908de0247c715423; no voters: 05aa1d36d28a48138040f6c78c4b34a5, ec8c86cff51f48bf9ecc5177e6266b7b
I20260807 03:37:29.277957 16694 raft_consensus.cc:2726] T 242a724d826340b693e5b0767a1ba28c P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:37:29.278476 16687 raft_consensus.cc:374] T fe184e38bf0d4e51bf42238d161ac738 P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:29.279183 16733 raft_consensus.cc:2726] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260807 03:37:29.265441 16433 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:37775, user_credentials={real_user=slave}} blocked reactor thread for 108875us
I20260807 03:37:29.281602 16365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e409bda14f142959c8127f3bd2643fc" candidate_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ca3bab29b4c4dce908de0247c715423" is_pre_election: true
I20260807 03:37:29.283231 16365 raft_consensus.cc:2454] T 5e409bda14f142959c8127f3bd2643fc P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec8c86cff51f48bf9ecc5177e6266b7b in term 0.
I20260807 03:37:29.279153 16687 raft_consensus.cc:729] T fe184e38bf0d4e51bf42238d161ac738 P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ca3bab29b4c4dce908de0247c715423, State: Initialized, Role: FOLLOWER
I20260807 03:37:29.287103 16479 raft_consensus.cc:2454] T fe184e38bf0d4e51bf42238d161ac738 P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec8c86cff51f48bf9ecc5177e6266b7b in term 1.
W20260807 03:37:29.282817 16556 leader_election.cc:341] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129:36375): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:37:29.288678 16687 consensus_queue.cc:260] T fe184e38bf0d4e51bf42238d161ac738 P 9ca3bab29b4c4dce908de0247c715423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
W20260807 03:37:29.289837 16433 leader_election.cc:341] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131:37775): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:37:29.290249 16557 leader_election.cc:302] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05aa1d36d28a48138040f6c78c4b34a5, ec8c86cff51f48bf9ecc5177e6266b7b; no voters: 9ca3bab29b4c4dce908de0247c715423
I20260807 03:37:29.295439 16687 ts_tablet_manager.cc:1428] T fe184e38bf0d4e51bf42238d161ac738 P 9ca3bab29b4c4dce908de0247c715423: Time spent starting tablet: real 0.022s user 0.005s sys 0.004s
I20260807 03:37:29.292320 16739 raft_consensus.cc:2781] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:37:29.293829 16484 raft_consensus.cc:2454] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ca3bab29b4c4dce908de0247c715423 in term 1.
I20260807 03:37:29.291944 16433 leader_election.cc:302] T a4e0d750de56427e8a0af5bc030271c4 P 05aa1d36d28a48138040f6c78c4b34a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05aa1d36d28a48138040f6c78c4b34a5, ec8c86cff51f48bf9ecc5177e6266b7b; no voters: 9ca3bab29b4c4dce908de0247c715423
I20260807 03:37:29.299026 16314 leader_election.cc:302] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05aa1d36d28a48138040f6c78c4b34a5, 9ca3bab29b4c4dce908de0247c715423; no voters: ec8c86cff51f48bf9ecc5177e6266b7b
I20260807 03:37:29.296749 16739 raft_consensus.cc:686] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b [term 1 LEADER]: Becoming Leader. State: Replica: ec8c86cff51f48bf9ecc5177e6266b7b, State: Running, Role: LEADER
I20260807 03:37:29.299911 16733 raft_consensus.cc:2781] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:37:29.300374 16733 raft_consensus.cc:686] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [term 1 LEADER]: Becoming Leader. State: Replica: 9ca3bab29b4c4dce908de0247c715423, State: Running, Role: LEADER
I20260807 03:37:29.297322 16686 tablet_bootstrap.cc:492] T d07d00a26d54438392d5db566a17627b P ec8c86cff51f48bf9ecc5177e6266b7b: No bootstrap required, opened a new log
I20260807 03:37:29.300704 16739 consensus_queue.cc:237] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.301322 16733 consensus_queue.cc:237] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.301190 16736 raft_consensus.cc:2781] T a4e0d750de56427e8a0af5bc030271c4 P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:37:29.289084 16556 leader_election.cc:302] T 5e409bda14f142959c8127f3bd2643fc P ec8c86cff51f48bf9ecc5177e6266b7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ca3bab29b4c4dce908de0247c715423, ec8c86cff51f48bf9ecc5177e6266b7b; no voters: 05aa1d36d28a48138040f6c78c4b34a5
I20260807 03:37:29.302641 16736 raft_consensus.cc:482] T a4e0d750de56427e8a0af5bc030271c4 P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:37:29.293200 16607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e0fd324ec84973b604e22e851d7b49" candidate_uuid: "05aa1d36d28a48138040f6c78c4b34a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b"
I20260807 03:37:29.302940 16696 raft_consensus.cc:2781] T 5e409bda14f142959c8127f3bd2643fc P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:37:29.303081 16736 raft_consensus.cc:3037] T a4e0d750de56427e8a0af5bc030271c4 P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:37:29.303392 16696 raft_consensus.cc:482] T 5e409bda14f142959c8127f3bd2643fc P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:37:29.302114 16686 ts_tablet_manager.cc:1397] T d07d00a26d54438392d5db566a17627b P ec8c86cff51f48bf9ecc5177e6266b7b: Time spent bootstrapping tablet: real 0.222s user 0.042s sys 0.046s
W20260807 03:37:29.304103 16433 leader_election.cc:341] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131:37775): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:37:29.304538 16696 raft_consensus.cc:3037] T 5e409bda14f142959c8127f3bd2643fc P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:37:29.304713 16433 leader_election.cc:302] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 05aa1d36d28a48138040f6c78c4b34a5; no voters: 9ca3bab29b4c4dce908de0247c715423, ec8c86cff51f48bf9ecc5177e6266b7b
I20260807 03:37:29.305992 16741 raft_consensus.cc:2726] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260807 03:37:29.309820 16687 tablet_bootstrap.cc:492] T 242a724d826340b693e5b0767a1ba28c P 9ca3bab29b4c4dce908de0247c715423: Bootstrap starting.
I20260807 03:37:29.313336 16686 raft_consensus.cc:348] T d07d00a26d54438392d5db566a17627b P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.314373 16686 raft_consensus.cc:374] T d07d00a26d54438392d5db566a17627b P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:29.315403 16686 raft_consensus.cc:729] T d07d00a26d54438392d5db566a17627b P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec8c86cff51f48bf9ecc5177e6266b7b, State: Initialized, Role: FOLLOWER
I20260807 03:37:29.316481 16684 tablet_bootstrap.cc:654] T 5e409bda14f142959c8127f3bd2643fc P 05aa1d36d28a48138040f6c78c4b34a5: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:29.318639 16686 consensus_queue.cc:260] T d07d00a26d54438392d5db566a17627b P ec8c86cff51f48bf9ecc5177e6266b7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.324203 16736 raft_consensus.cc:504] T a4e0d750de56427e8a0af5bc030271c4 P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.332207 16696 raft_consensus.cc:504] T 5e409bda14f142959c8127f3bd2643fc P ec8c86cff51f48bf9ecc5177e6266b7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.333458 16736 leader_election.cc:288] T a4e0d750de56427e8a0af5bc030271c4 P 05aa1d36d28a48138040f6c78c4b34a5 [CANDIDATE]: Term 1 election: Requested vote from peers 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129:36375), ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131:37775)
I20260807 03:37:29.335259 16684 tablet_bootstrap.cc:492] T 5e409bda14f142959c8127f3bd2643fc P 05aa1d36d28a48138040f6c78c4b34a5: No bootstrap required, opened a new log
I20260807 03:37:29.336500 16684 ts_tablet_manager.cc:1397] T 5e409bda14f142959c8127f3bd2643fc P 05aa1d36d28a48138040f6c78c4b34a5: Time spent bootstrapping tablet: real 0.068s user 0.014s sys 0.002s
I20260807 03:37:29.327965 16687 tablet_bootstrap.cc:654] T 242a724d826340b693e5b0767a1ba28c P 9ca3bab29b4c4dce908de0247c715423: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:29.333722 16607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4e0d750de56427e8a0af5bc030271c4" candidate_uuid: "05aa1d36d28a48138040f6c78c4b34a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b"
I20260807 03:37:29.337463 16607 raft_consensus.cc:3037] T a4e0d750de56427e8a0af5bc030271c4 P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:37:29.338322 16367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e409bda14f142959c8127f3bd2643fc" candidate_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ca3bab29b4c4dce908de0247c715423"
I20260807 03:37:29.338922 16367 raft_consensus.cc:3037] T 5e409bda14f142959c8127f3bd2643fc P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:37:29.333606 16365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4e0d750de56427e8a0af5bc030271c4" candidate_uuid: "05aa1d36d28a48138040f6c78c4b34a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ca3bab29b4c4dce908de0247c715423"
W20260807 03:37:29.340462 16432 leader_election.cc:341] T a4e0d750de56427e8a0af5bc030271c4 P 05aa1d36d28a48138040f6c78c4b34a5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129:36375): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:37:29.345105 16607 raft_consensus.cc:2454] T a4e0d750de56427e8a0af5bc030271c4 P ec8c86cff51f48bf9ecc5177e6266b7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05aa1d36d28a48138040f6c78c4b34a5 in term 1.
I20260807 03:37:29.346366 16433 leader_election.cc:302] T a4e0d750de56427e8a0af5bc030271c4 P 05aa1d36d28a48138040f6c78c4b34a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05aa1d36d28a48138040f6c78c4b34a5, ec8c86cff51f48bf9ecc5177e6266b7b; no voters: 9ca3bab29b4c4dce908de0247c715423
I20260807 03:37:29.347594 16686 ts_tablet_manager.cc:1428] T d07d00a26d54438392d5db566a17627b P ec8c86cff51f48bf9ecc5177e6266b7b: Time spent starting tablet: real 0.043s user 0.009s sys 0.002s
I20260807 03:37:29.348392 16367 raft_consensus.cc:2454] T 5e409bda14f142959c8127f3bd2643fc P 9ca3bab29b4c4dce908de0247c715423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec8c86cff51f48bf9ecc5177e6266b7b in term 1.
I20260807 03:37:29.348201 16736 raft_consensus.cc:2781] T a4e0d750de56427e8a0af5bc030271c4 P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:37:29.349130 16736 raft_consensus.cc:686] T a4e0d750de56427e8a0af5bc030271c4 P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 LEADER]: Becoming Leader. State: Replica: 05aa1d36d28a48138040f6c78c4b34a5, State: Running, Role: LEADER
I20260807 03:37:29.349538 16556 leader_election.cc:302] T 5e409bda14f142959c8127f3bd2643fc P ec8c86cff51f48bf9ecc5177e6266b7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ca3bab29b4c4dce908de0247c715423, ec8c86cff51f48bf9ecc5177e6266b7b; no voters:
I20260807 03:37:29.347656 16206 catalog_manager.cc:5495] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b reported cstate change: term changed from 0 to 1, leader changed from <none> to ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131). New cstate: current_term: 1 leader_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } health_report { overall_health: HEALTHY } } }
I20260807 03:37:29.350431 16750 raft_consensus.cc:2781] T 5e409bda14f142959c8127f3bd2643fc P ec8c86cff51f48bf9ecc5177e6266b7b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:37:29.350123 16736 consensus_queue.cc:237] T a4e0d750de56427e8a0af5bc030271c4 P 05aa1d36d28a48138040f6c78c4b34a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.353012 16684 raft_consensus.cc:348] T 5e409bda14f142959c8127f3bd2643fc P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.353770 16684 raft_consensus.cc:374] T 5e409bda14f142959c8127f3bd2643fc P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:29.354126 16684 raft_consensus.cc:729] T 5e409bda14f142959c8127f3bd2643fc P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05aa1d36d28a48138040f6c78c4b34a5, State: Initialized, Role: FOLLOWER
I20260807 03:37:29.357092 16686 tablet_bootstrap.cc:492] T f0191ce545e0490986331a9c562b056d P ec8c86cff51f48bf9ecc5177e6266b7b: Bootstrap starting.
I20260807 03:37:29.362236 16687 tablet_bootstrap.cc:492] T 242a724d826340b693e5b0767a1ba28c P 9ca3bab29b4c4dce908de0247c715423: No bootstrap required, opened a new log
I20260807 03:37:29.362775 16687 ts_tablet_manager.cc:1397] T 242a724d826340b693e5b0767a1ba28c P 9ca3bab29b4c4dce908de0247c715423: Time spent bootstrapping tablet: real 0.055s user 0.013s sys 0.008s
I20260807 03:37:29.365463 16484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e409bda14f142959c8127f3bd2643fc" candidate_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05aa1d36d28a48138040f6c78c4b34a5"
I20260807 03:37:29.362752 16696 leader_election.cc:288] T 5e409bda14f142959c8127f3bd2643fc P ec8c86cff51f48bf9ecc5177e6266b7b [CANDIDATE]: Term 1 election: Requested vote from peers 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129:36375), 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130:39359)
I20260807 03:37:29.366854 16750 raft_consensus.cc:686] T 5e409bda14f142959c8127f3bd2643fc P ec8c86cff51f48bf9ecc5177e6266b7b [term 1 LEADER]: Becoming Leader. State: Replica: ec8c86cff51f48bf9ecc5177e6266b7b, State: Running, Role: LEADER
I20260807 03:37:29.366496 16687 raft_consensus.cc:348] T 242a724d826340b693e5b0767a1ba28c P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.367401 16687 raft_consensus.cc:374] T 242a724d826340b693e5b0767a1ba28c P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:29.367748 16687 raft_consensus.cc:729] T 242a724d826340b693e5b0767a1ba28c P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ca3bab29b4c4dce908de0247c715423, State: Initialized, Role: FOLLOWER
I20260807 03:37:29.355064 16684 consensus_queue.cc:260] T 5e409bda14f142959c8127f3bd2643fc P 05aa1d36d28a48138040f6c78c4b34a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.367690 16750 consensus_queue.cc:237] T 5e409bda14f142959c8127f3bd2643fc P ec8c86cff51f48bf9ecc5177e6266b7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.368801 16687 consensus_queue.cc:260] T 242a724d826340b693e5b0767a1ba28c P 9ca3bab29b4c4dce908de0247c715423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.369159 16207 catalog_manager.cc:5495] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129). New cstate: current_term: 1 leader_uuid: "9ca3bab29b4c4dce908de0247c715423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } health_report { overall_health: UNKNOWN } } }
I20260807 03:37:29.370335 16684 ts_tablet_manager.cc:1428] T 5e409bda14f142959c8127f3bd2643fc P 05aa1d36d28a48138040f6c78c4b34a5: Time spent starting tablet: real 0.034s user 0.007s sys 0.000s
I20260807 03:37:29.371822 16484 raft_consensus.cc:3037] T 5e409bda14f142959c8127f3bd2643fc P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:37:29.371325 16684 tablet_bootstrap.cc:492] T 485e8d9a337d4e4f9207f737e3a8812b P 05aa1d36d28a48138040f6c78c4b34a5: Bootstrap starting.
I20260807 03:37:29.372159 16687 ts_tablet_manager.cc:1428] T 242a724d826340b693e5b0767a1ba28c P 9ca3bab29b4c4dce908de0247c715423: Time spent starting tablet: real 0.009s user 0.006s sys 0.002s
I20260807 03:37:29.377249 16687 tablet_bootstrap.cc:492] T d07d00a26d54438392d5db566a17627b P 9ca3bab29b4c4dce908de0247c715423: Bootstrap starting.
I20260807 03:37:29.376466 16208 catalog_manager.cc:5495] T a4e0d750de56427e8a0af5bc030271c4 P 05aa1d36d28a48138040f6c78c4b34a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130). New cstate: current_term: 1 leader_uuid: "05aa1d36d28a48138040f6c78c4b34a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } health_report { overall_health: UNKNOWN } } }
I20260807 03:37:29.377971 16686 tablet_bootstrap.cc:654] T f0191ce545e0490986331a9c562b056d P ec8c86cff51f48bf9ecc5177e6266b7b: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:29.391873 16484 raft_consensus.cc:2454] T 5e409bda14f142959c8127f3bd2643fc P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec8c86cff51f48bf9ecc5177e6266b7b in term 1.
I20260807 03:37:29.404325 16686 tablet_bootstrap.cc:492] T f0191ce545e0490986331a9c562b056d P ec8c86cff51f48bf9ecc5177e6266b7b: No bootstrap required, opened a new log
I20260807 03:37:29.404459 16684 tablet_bootstrap.cc:654] T 485e8d9a337d4e4f9207f737e3a8812b P 05aa1d36d28a48138040f6c78c4b34a5: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:29.405071 16686 ts_tablet_manager.cc:1397] T f0191ce545e0490986331a9c562b056d P ec8c86cff51f48bf9ecc5177e6266b7b: Time spent bootstrapping tablet: real 0.048s user 0.011s sys 0.005s
I20260807 03:37:29.408555 16686 raft_consensus.cc:348] T f0191ce545e0490986331a9c562b056d P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.409737 16686 raft_consensus.cc:374] T f0191ce545e0490986331a9c562b056d P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:29.410219 16686 raft_consensus.cc:729] T f0191ce545e0490986331a9c562b056d P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec8c86cff51f48bf9ecc5177e6266b7b, State: Initialized, Role: FOLLOWER
I20260807 03:37:29.411264 16686 consensus_queue.cc:260] T f0191ce545e0490986331a9c562b056d P ec8c86cff51f48bf9ecc5177e6266b7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.413617 16207 catalog_manager.cc:5495] T 5e409bda14f142959c8127f3bd2643fc P ec8c86cff51f48bf9ecc5177e6266b7b reported cstate change: term changed from 0 to 1, leader changed from <none> to ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131). New cstate: current_term: 1 leader_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } health_report { overall_health: HEALTHY } } }
I20260807 03:37:29.414670 16686 ts_tablet_manager.cc:1428] T f0191ce545e0490986331a9c562b056d P ec8c86cff51f48bf9ecc5177e6266b7b: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260807 03:37:29.416273 16686 tablet_bootstrap.cc:492] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b: Bootstrap starting.
I20260807 03:37:29.418493 16736 raft_consensus.cc:482] T d07d00a26d54438392d5db566a17627b P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:37:29.418941 16736 raft_consensus.cc:504] T d07d00a26d54438392d5db566a17627b P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.421027 16736 leader_election.cc:288] T d07d00a26d54438392d5db566a17627b P 05aa1d36d28a48138040f6c78c4b34a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129:36375), ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131:37775)
I20260807 03:37:29.421602 16687 tablet_bootstrap.cc:654] T d07d00a26d54438392d5db566a17627b P 9ca3bab29b4c4dce908de0247c715423: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:29.421818 16367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d07d00a26d54438392d5db566a17627b" candidate_uuid: "05aa1d36d28a48138040f6c78c4b34a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ca3bab29b4c4dce908de0247c715423" is_pre_election: true
I20260807 03:37:29.422768 16684 tablet_bootstrap.cc:492] T 485e8d9a337d4e4f9207f737e3a8812b P 05aa1d36d28a48138040f6c78c4b34a5: No bootstrap required, opened a new log
I20260807 03:37:29.422948 16607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d07d00a26d54438392d5db566a17627b" candidate_uuid: "05aa1d36d28a48138040f6c78c4b34a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" is_pre_election: true
I20260807 03:37:29.423280 16684 ts_tablet_manager.cc:1397] T 485e8d9a337d4e4f9207f737e3a8812b P 05aa1d36d28a48138040f6c78c4b34a5: Time spent bootstrapping tablet: real 0.052s user 0.015s sys 0.007s
I20260807 03:37:29.423580 16607 raft_consensus.cc:2454] T d07d00a26d54438392d5db566a17627b P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05aa1d36d28a48138040f6c78c4b34a5 in term 0.
W20260807 03:37:29.424661 16432 leader_election.cc:341] T d07d00a26d54438392d5db566a17627b P 05aa1d36d28a48138040f6c78c4b34a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129:36375): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:37:29.426254 16433 leader_election.cc:302] T d07d00a26d54438392d5db566a17627b P 05aa1d36d28a48138040f6c78c4b34a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05aa1d36d28a48138040f6c78c4b34a5, ec8c86cff51f48bf9ecc5177e6266b7b; no voters: 9ca3bab29b4c4dce908de0247c715423
I20260807 03:37:29.426616 16684 raft_consensus.cc:348] T 485e8d9a337d4e4f9207f737e3a8812b P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.427312 16736 raft_consensus.cc:2781] T d07d00a26d54438392d5db566a17627b P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:37:29.427531 16684 raft_consensus.cc:374] T 485e8d9a337d4e4f9207f737e3a8812b P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:29.427670 16736 raft_consensus.cc:482] T d07d00a26d54438392d5db566a17627b P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:37:29.427928 16684 raft_consensus.cc:729] T 485e8d9a337d4e4f9207f737e3a8812b P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05aa1d36d28a48138040f6c78c4b34a5, State: Initialized, Role: FOLLOWER
I20260807 03:37:29.428081 16736 raft_consensus.cc:3037] T d07d00a26d54438392d5db566a17627b P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:37:29.428862 16684 consensus_queue.cc:260] T 485e8d9a337d4e4f9207f737e3a8812b P 05aa1d36d28a48138040f6c78c4b34a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.431233 16684 ts_tablet_manager.cc:1428] T 485e8d9a337d4e4f9207f737e3a8812b P 05aa1d36d28a48138040f6c78c4b34a5: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20260807 03:37:29.433918 16686 tablet_bootstrap.cc:654] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:29.435022 16736 raft_consensus.cc:504] T d07d00a26d54438392d5db566a17627b P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.438150 16367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d07d00a26d54438392d5db566a17627b" candidate_uuid: "05aa1d36d28a48138040f6c78c4b34a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ca3bab29b4c4dce908de0247c715423"
W20260807 03:37:29.439810 16432 leader_election.cc:341] T d07d00a26d54438392d5db566a17627b P 05aa1d36d28a48138040f6c78c4b34a5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129:36375): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:37:29.442543 16607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d07d00a26d54438392d5db566a17627b" candidate_uuid: "05aa1d36d28a48138040f6c78c4b34a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b"
I20260807 03:37:29.443296 16607 raft_consensus.cc:3037] T d07d00a26d54438392d5db566a17627b P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:37:29.432054 16684 tablet_bootstrap.cc:492] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5: Bootstrap starting.
I20260807 03:37:29.452205 16736 leader_election.cc:288] T d07d00a26d54438392d5db566a17627b P 05aa1d36d28a48138040f6c78c4b34a5 [CANDIDATE]: Term 1 election: Requested vote from peers 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129:36375), ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131:37775)
I20260807 03:37:29.452939 16607 raft_consensus.cc:2454] T d07d00a26d54438392d5db566a17627b P ec8c86cff51f48bf9ecc5177e6266b7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05aa1d36d28a48138040f6c78c4b34a5 in term 1.
I20260807 03:37:29.454216 16433 leader_election.cc:302] T d07d00a26d54438392d5db566a17627b P 05aa1d36d28a48138040f6c78c4b34a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05aa1d36d28a48138040f6c78c4b34a5, ec8c86cff51f48bf9ecc5177e6266b7b; no voters: 9ca3bab29b4c4dce908de0247c715423
I20260807 03:37:29.455582 16736 raft_consensus.cc:2781] T d07d00a26d54438392d5db566a17627b P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:37:29.455754 16684 tablet_bootstrap.cc:654] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:29.456245 16736 raft_consensus.cc:686] T d07d00a26d54438392d5db566a17627b P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 LEADER]: Becoming Leader. State: Replica: 05aa1d36d28a48138040f6c78c4b34a5, State: Running, Role: LEADER
I20260807 03:37:29.457288 16736 consensus_queue.cc:237] T d07d00a26d54438392d5db566a17627b P 05aa1d36d28a48138040f6c78c4b34a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.473604 16687 tablet_bootstrap.cc:492] T d07d00a26d54438392d5db566a17627b P 9ca3bab29b4c4dce908de0247c715423: No bootstrap required, opened a new log
I20260807 03:37:29.473929 16686 tablet_bootstrap.cc:492] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b: No bootstrap required, opened a new log
I20260807 03:37:29.474097 16687 ts_tablet_manager.cc:1397] T d07d00a26d54438392d5db566a17627b P 9ca3bab29b4c4dce908de0247c715423: Time spent bootstrapping tablet: real 0.097s user 0.018s sys 0.022s
I20260807 03:37:29.474622 16686 ts_tablet_manager.cc:1397] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b: Time spent bootstrapping tablet: real 0.059s user 0.030s sys 0.016s
I20260807 03:37:29.477097 16687 raft_consensus.cc:348] T d07d00a26d54438392d5db566a17627b P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.477784 16684 tablet_bootstrap.cc:492] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5: No bootstrap required, opened a new log
I20260807 03:37:29.477914 16687 raft_consensus.cc:374] T d07d00a26d54438392d5db566a17627b P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:29.478367 16684 ts_tablet_manager.cc:1397] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5: Time spent bootstrapping tablet: real 0.046s user 0.007s sys 0.007s
I20260807 03:37:29.478364 16687 raft_consensus.cc:729] T d07d00a26d54438392d5db566a17627b P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ca3bab29b4c4dce908de0247c715423, State: Initialized, Role: FOLLOWER
I20260807 03:37:29.477701 16686 raft_consensus.cc:348] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.479396 16686 raft_consensus.cc:374] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:29.479228 16687 consensus_queue.cc:260] T d07d00a26d54438392d5db566a17627b P 9ca3bab29b4c4dce908de0247c715423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.479764 16686 raft_consensus.cc:729] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec8c86cff51f48bf9ecc5177e6266b7b, State: Initialized, Role: FOLLOWER
I20260807 03:37:29.480640 16686 consensus_queue.cc:260] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.481218 16684 raft_consensus.cc:348] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.481834 16684 raft_consensus.cc:374] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:29.482093 16684 raft_consensus.cc:729] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05aa1d36d28a48138040f6c78c4b34a5, State: Initialized, Role: FOLLOWER
I20260807 03:37:29.482748 16684 consensus_queue.cc:260] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.483264 16686 ts_tablet_manager.cc:1428] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b: Time spent starting tablet: real 0.008s user 0.004s sys 0.003s
I20260807 03:37:29.484325 16686 tablet_bootstrap.cc:492] T 10452db9b82745b68c6c353b25eff5ef P ec8c86cff51f48bf9ecc5177e6266b7b: Bootstrap starting.
I20260807 03:37:29.484670 16684 ts_tablet_manager.cc:1428] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20260807 03:37:29.490561 16686 tablet_bootstrap.cc:654] T 10452db9b82745b68c6c353b25eff5ef P ec8c86cff51f48bf9ecc5177e6266b7b: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:29.498911 16687 ts_tablet_manager.cc:1428] T d07d00a26d54438392d5db566a17627b P 9ca3bab29b4c4dce908de0247c715423: Time spent starting tablet: real 0.024s user 0.006s sys 0.002s
I20260807 03:37:29.499845 16687 tablet_bootstrap.cc:492] T a4e0d750de56427e8a0af5bc030271c4 P 9ca3bab29b4c4dce908de0247c715423: Bootstrap starting.
I20260807 03:37:29.507356 16206 catalog_manager.cc:5495] T d07d00a26d54438392d5db566a17627b P 05aa1d36d28a48138040f6c78c4b34a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130). New cstate: current_term: 1 leader_uuid: "05aa1d36d28a48138040f6c78c4b34a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } health_report { overall_health: UNKNOWN } } }
W20260807 03:37:29.508838 16279 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 03:37:29.507364 16687 tablet_bootstrap.cc:654] T a4e0d750de56427e8a0af5bc030271c4 P 9ca3bab29b4c4dce908de0247c715423: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:29.511925 16686 tablet_bootstrap.cc:492] T 10452db9b82745b68c6c353b25eff5ef P ec8c86cff51f48bf9ecc5177e6266b7b: No bootstrap required, opened a new log
I20260807 03:37:29.512384 16686 ts_tablet_manager.cc:1397] T 10452db9b82745b68c6c353b25eff5ef P ec8c86cff51f48bf9ecc5177e6266b7b: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.010s
I20260807 03:37:29.514786 16686 raft_consensus.cc:348] T 10452db9b82745b68c6c353b25eff5ef P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.515421 16686 raft_consensus.cc:374] T 10452db9b82745b68c6c353b25eff5ef P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:29.515650 16686 raft_consensus.cc:729] T 10452db9b82745b68c6c353b25eff5ef P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec8c86cff51f48bf9ecc5177e6266b7b, State: Initialized, Role: FOLLOWER
I20260807 03:37:29.516358 16686 consensus_queue.cc:260] T 10452db9b82745b68c6c353b25eff5ef P ec8c86cff51f48bf9ecc5177e6266b7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.518486 16686 ts_tablet_manager.cc:1428] T 10452db9b82745b68c6c353b25eff5ef P ec8c86cff51f48bf9ecc5177e6266b7b: Time spent starting tablet: real 0.006s user 0.002s sys 0.002s
I20260807 03:37:29.519377 16686 tablet_bootstrap.cc:492] T 242a724d826340b693e5b0767a1ba28c P ec8c86cff51f48bf9ecc5177e6266b7b: Bootstrap starting.
I20260807 03:37:29.526420 16686 tablet_bootstrap.cc:654] T 242a724d826340b693e5b0767a1ba28c P ec8c86cff51f48bf9ecc5177e6266b7b: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:29.533699 16687 tablet_bootstrap.cc:492] T a4e0d750de56427e8a0af5bc030271c4 P 9ca3bab29b4c4dce908de0247c715423: No bootstrap required, opened a new log
I20260807 03:37:29.534137 16687 ts_tablet_manager.cc:1397] T a4e0d750de56427e8a0af5bc030271c4 P 9ca3bab29b4c4dce908de0247c715423: Time spent bootstrapping tablet: real 0.035s user 0.006s sys 0.015s
I20260807 03:37:29.536536 16687 raft_consensus.cc:348] T a4e0d750de56427e8a0af5bc030271c4 P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.537150 16687 raft_consensus.cc:374] T a4e0d750de56427e8a0af5bc030271c4 P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:29.537402 16687 raft_consensus.cc:729] T a4e0d750de56427e8a0af5bc030271c4 P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ca3bab29b4c4dce908de0247c715423, State: Initialized, Role: FOLLOWER
I20260807 03:37:29.538992 16687 consensus_queue.cc:260] T a4e0d750de56427e8a0af5bc030271c4 P 9ca3bab29b4c4dce908de0247c715423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.540020 16686 tablet_bootstrap.cc:492] T 242a724d826340b693e5b0767a1ba28c P ec8c86cff51f48bf9ecc5177e6266b7b: No bootstrap required, opened a new log
I20260807 03:37:29.540428 16686 ts_tablet_manager.cc:1397] T 242a724d826340b693e5b0767a1ba28c P ec8c86cff51f48bf9ecc5177e6266b7b: Time spent bootstrapping tablet: real 0.021s user 0.016s sys 0.003s
I20260807 03:37:29.541028 16687 ts_tablet_manager.cc:1428] T a4e0d750de56427e8a0af5bc030271c4 P 9ca3bab29b4c4dce908de0247c715423: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20260807 03:37:29.542554 16686 raft_consensus.cc:348] T 242a724d826340b693e5b0767a1ba28c P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.543231 16686 raft_consensus.cc:374] T 242a724d826340b693e5b0767a1ba28c P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:29.543454 16686 raft_consensus.cc:729] T 242a724d826340b693e5b0767a1ba28c P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec8c86cff51f48bf9ecc5177e6266b7b, State: Initialized, Role: FOLLOWER
I20260807 03:37:29.543983 16686 consensus_queue.cc:260] T 242a724d826340b693e5b0767a1ba28c P ec8c86cff51f48bf9ecc5177e6266b7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.545836 16686 ts_tablet_manager.cc:1428] T 242a724d826340b693e5b0767a1ba28c P ec8c86cff51f48bf9ecc5177e6266b7b: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260807 03:37:29.547920 16733 raft_consensus.cc:482] T fe184e38bf0d4e51bf42238d161ac738 P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:37:29.548372 16733 raft_consensus.cc:504] T fe184e38bf0d4e51bf42238d161ac738 P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.549870 16733 leader_election.cc:288] T fe184e38bf0d4e51bf42238d161ac738 P 9ca3bab29b4c4dce908de0247c715423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130:39359), ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131:37775)
I20260807 03:37:29.550952 16607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe184e38bf0d4e51bf42238d161ac738" candidate_uuid: "9ca3bab29b4c4dce908de0247c715423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" is_pre_election: true
I20260807 03:37:29.552513 16750 raft_consensus.cc:482] T 242a724d826340b693e5b0767a1ba28c P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:37:29.552973 16484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe184e38bf0d4e51bf42238d161ac738" candidate_uuid: "9ca3bab29b4c4dce908de0247c715423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05aa1d36d28a48138040f6c78c4b34a5" is_pre_election: true
I20260807 03:37:29.552906 16750 raft_consensus.cc:504] T 242a724d826340b693e5b0767a1ba28c P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.553704 16484 raft_consensus.cc:2379] T fe184e38bf0d4e51bf42238d161ac738 P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9ca3bab29b4c4dce908de0247c715423 in current term 1: Already voted for candidate ec8c86cff51f48bf9ecc5177e6266b7b in this term.
I20260807 03:37:29.554564 16314 leader_election.cc:302] T fe184e38bf0d4e51bf42238d161ac738 P 9ca3bab29b4c4dce908de0247c715423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ca3bab29b4c4dce908de0247c715423; no voters: 05aa1d36d28a48138040f6c78c4b34a5, ec8c86cff51f48bf9ecc5177e6266b7b
I20260807 03:37:29.554777 16750 leader_election.cc:288] T 242a724d826340b693e5b0767a1ba28c P ec8c86cff51f48bf9ecc5177e6266b7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129:36375), 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130:39359)
I20260807 03:37:29.555338 16733 raft_consensus.cc:3037] T fe184e38bf0d4e51bf42238d161ac738 P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:37:29.556250 16367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242a724d826340b693e5b0767a1ba28c" candidate_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ca3bab29b4c4dce908de0247c715423" is_pre_election: true
I20260807 03:37:29.556872 16367 raft_consensus.cc:2454] T 242a724d826340b693e5b0767a1ba28c P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec8c86cff51f48bf9ecc5177e6266b7b in term 0.
I20260807 03:37:29.556982 16484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242a724d826340b693e5b0767a1ba28c" candidate_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05aa1d36d28a48138040f6c78c4b34a5" is_pre_election: true
I20260807 03:37:29.557564 16484 raft_consensus.cc:2454] T 242a724d826340b693e5b0767a1ba28c P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec8c86cff51f48bf9ecc5177e6266b7b in term 0.
I20260807 03:37:29.558552 16557 leader_election.cc:302] T 242a724d826340b693e5b0767a1ba28c P ec8c86cff51f48bf9ecc5177e6266b7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05aa1d36d28a48138040f6c78c4b34a5, ec8c86cff51f48bf9ecc5177e6266b7b; no voters:
I20260807 03:37:29.559348 16750 raft_consensus.cc:2781] T 242a724d826340b693e5b0767a1ba28c P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:37:29.559720 16750 raft_consensus.cc:482] T 242a724d826340b693e5b0767a1ba28c P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:37:29.560016 16750 raft_consensus.cc:3037] T 242a724d826340b693e5b0767a1ba28c P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:37:29.561910 16733 raft_consensus.cc:2726] T fe184e38bf0d4e51bf42238d161ac738 P 9ca3bab29b4c4dce908de0247c715423 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:37:29.565330 16750 raft_consensus.cc:504] T 242a724d826340b693e5b0767a1ba28c P ec8c86cff51f48bf9ecc5177e6266b7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.567667 16367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242a724d826340b693e5b0767a1ba28c" candidate_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ca3bab29b4c4dce908de0247c715423"
I20260807 03:37:29.568261 16367 raft_consensus.cc:3037] T 242a724d826340b693e5b0767a1ba28c P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:37:29.568217 16484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242a724d826340b693e5b0767a1ba28c" candidate_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05aa1d36d28a48138040f6c78c4b34a5"
I20260807 03:37:29.568877 16484 raft_consensus.cc:3037] T 242a724d826340b693e5b0767a1ba28c P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:37:29.575801 16367 raft_consensus.cc:2454] T 242a724d826340b693e5b0767a1ba28c P 9ca3bab29b4c4dce908de0247c715423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec8c86cff51f48bf9ecc5177e6266b7b in term 1.
I20260807 03:37:29.576826 16556 leader_election.cc:302] T 242a724d826340b693e5b0767a1ba28c P ec8c86cff51f48bf9ecc5177e6266b7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ca3bab29b4c4dce908de0247c715423, ec8c86cff51f48bf9ecc5177e6266b7b; no voters:
I20260807 03:37:29.576942 16484 raft_consensus.cc:2454] T 242a724d826340b693e5b0767a1ba28c P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec8c86cff51f48bf9ecc5177e6266b7b in term 1.
I20260807 03:37:29.577795 16754 raft_consensus.cc:2781] T 242a724d826340b693e5b0767a1ba28c P ec8c86cff51f48bf9ecc5177e6266b7b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:37:29.578326 16754 raft_consensus.cc:686] T 242a724d826340b693e5b0767a1ba28c P ec8c86cff51f48bf9ecc5177e6266b7b [term 1 LEADER]: Becoming Leader. State: Replica: ec8c86cff51f48bf9ecc5177e6266b7b, State: Running, Role: LEADER
I20260807 03:37:29.579042 16754 consensus_queue.cc:237] T 242a724d826340b693e5b0767a1ba28c P ec8c86cff51f48bf9ecc5177e6266b7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.574462 16750 leader_election.cc:288] T 242a724d826340b693e5b0767a1ba28c P ec8c86cff51f48bf9ecc5177e6266b7b [CANDIDATE]: Term 1 election: Requested vote from peers 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129:36375), 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130:39359)
I20260807 03:37:29.586733 16206 catalog_manager.cc:5495] T 242a724d826340b693e5b0767a1ba28c P ec8c86cff51f48bf9ecc5177e6266b7b reported cstate change: term changed from 0 to 1, leader changed from <none> to ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131). New cstate: current_term: 1 leader_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } health_report { overall_health: HEALTHY } } }
I20260807 03:37:29.596982 16741 raft_consensus.cc:482] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:37:29.597436 16741 raft_consensus.cc:504] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.598922 16741 leader_election.cc:288] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129:36375), ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131:37775)
I20260807 03:37:29.599861 16367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0191ce545e0490986331a9c562b056d" candidate_uuid: "05aa1d36d28a48138040f6c78c4b34a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ca3bab29b4c4dce908de0247c715423" is_pre_election: true
I20260807 03:37:29.600042 16607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0191ce545e0490986331a9c562b056d" candidate_uuid: "05aa1d36d28a48138040f6c78c4b34a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" is_pre_election: true
I20260807 03:37:29.600426 16367 raft_consensus.cc:2454] T f0191ce545e0490986331a9c562b056d P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05aa1d36d28a48138040f6c78c4b34a5 in term 0.
I20260807 03:37:29.600739 16607 raft_consensus.cc:2454] T f0191ce545e0490986331a9c562b056d P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05aa1d36d28a48138040f6c78c4b34a5 in term 0.
I20260807 03:37:29.601312 16432 leader_election.cc:302] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05aa1d36d28a48138040f6c78c4b34a5, 9ca3bab29b4c4dce908de0247c715423; no voters:
I20260807 03:37:29.602084 16741 raft_consensus.cc:2781] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:37:29.602401 16741 raft_consensus.cc:482] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:37:29.602696 16741 raft_consensus.cc:3037] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:37:29.607086 16741 raft_consensus.cc:504] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.608917 16741 leader_election.cc:288] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5 [CANDIDATE]: Term 1 election: Requested vote from peers 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129:36375), ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131:37775)
I20260807 03:37:29.609531 16607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0191ce545e0490986331a9c562b056d" candidate_uuid: "05aa1d36d28a48138040f6c78c4b34a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b"
I20260807 03:37:29.609421 16367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0191ce545e0490986331a9c562b056d" candidate_uuid: "05aa1d36d28a48138040f6c78c4b34a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ca3bab29b4c4dce908de0247c715423"
I20260807 03:37:29.610041 16607 raft_consensus.cc:3037] T f0191ce545e0490986331a9c562b056d P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:37:29.610100 16367 raft_consensus.cc:3037] T f0191ce545e0490986331a9c562b056d P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:37:29.616168 16367 raft_consensus.cc:2454] T f0191ce545e0490986331a9c562b056d P 9ca3bab29b4c4dce908de0247c715423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05aa1d36d28a48138040f6c78c4b34a5 in term 1.
I20260807 03:37:29.616168 16607 raft_consensus.cc:2454] T f0191ce545e0490986331a9c562b056d P ec8c86cff51f48bf9ecc5177e6266b7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05aa1d36d28a48138040f6c78c4b34a5 in term 1.
I20260807 03:37:29.617218 16432 leader_election.cc:302] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05aa1d36d28a48138040f6c78c4b34a5, 9ca3bab29b4c4dce908de0247c715423; no voters:
I20260807 03:37:29.617971 16741 raft_consensus.cc:2781] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:37:29.618561 16741 raft_consensus.cc:686] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 LEADER]: Becoming Leader. State: Replica: 05aa1d36d28a48138040f6c78c4b34a5, State: Running, Role: LEADER
I20260807 03:37:29.619246 16741 consensus_queue.cc:237] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.627022 16206 catalog_manager.cc:5495] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130). New cstate: current_term: 1 leader_uuid: "05aa1d36d28a48138040f6c78c4b34a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } health_report { overall_health: UNKNOWN } } }
I20260807 03:37:29.631246 16733 consensus_queue.cc:1035] T 485e8d9a337d4e4f9207f737e3a8812b P 9ca3bab29b4c4dce908de0247c715423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:37:29.652588 16484 raft_consensus.cc:3037] T 485e8d9a337d4e4f9207f737e3a8812b P 05aa1d36d28a48138040f6c78c4b34a5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:37:29.659703 16733 consensus_queue.cc:1035] T 485e8d9a337d4e4f9207f737e3a8812b P 9ca3bab29b4c4dce908de0247c715423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:37:29.721619 16754 consensus_queue.cc:1035] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:37:29.735535 16754 consensus_queue.cc:1035] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:37:29.751142 16754 raft_consensus.cc:482] T 10452db9b82745b68c6c353b25eff5ef P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:37:29.751662 16754 raft_consensus.cc:504] T 10452db9b82745b68c6c353b25eff5ef P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.753491 16754 leader_election.cc:288] T 10452db9b82745b68c6c353b25eff5ef P ec8c86cff51f48bf9ecc5177e6266b7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129:36375), 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130:39359)
I20260807 03:37:29.754175 16367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10452db9b82745b68c6c353b25eff5ef" candidate_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ca3bab29b4c4dce908de0247c715423" is_pre_election: true
I20260807 03:37:29.754468 16484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10452db9b82745b68c6c353b25eff5ef" candidate_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05aa1d36d28a48138040f6c78c4b34a5" is_pre_election: true
I20260807 03:37:29.755191 16484 raft_consensus.cc:2379] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ec8c86cff51f48bf9ecc5177e6266b7b in current term 1: Already voted for candidate 9ca3bab29b4c4dce908de0247c715423 in this term.
I20260807 03:37:29.756075 16557 leader_election.cc:302] T 10452db9b82745b68c6c353b25eff5ef P ec8c86cff51f48bf9ecc5177e6266b7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec8c86cff51f48bf9ecc5177e6266b7b; no voters: 05aa1d36d28a48138040f6c78c4b34a5, 9ca3bab29b4c4dce908de0247c715423
I20260807 03:37:29.756754 16754 raft_consensus.cc:3037] T 10452db9b82745b68c6c353b25eff5ef P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:37:29.763166 16754 raft_consensus.cc:2726] T 10452db9b82745b68c6c353b25eff5ef P ec8c86cff51f48bf9ecc5177e6266b7b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:37:29.822292 16733 consensus_queue.cc:1035] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:37:29.833860 16729 consensus_queue.cc:1035] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:37:29.844877 16754 consensus_queue.cc:1035] T 5e409bda14f142959c8127f3bd2643fc P ec8c86cff51f48bf9ecc5177e6266b7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:37:29.847977 16741 consensus_queue.cc:1035] T d07d00a26d54438392d5db566a17627b P 05aa1d36d28a48138040f6c78c4b34a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:37:29.851770 16729 raft_consensus.cc:482] T a4e0d750de56427e8a0af5bc030271c4 P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:37:29.852322 16729 raft_consensus.cc:504] T a4e0d750de56427e8a0af5bc030271c4 P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.854844 16729 leader_election.cc:288] T a4e0d750de56427e8a0af5bc030271c4 P 9ca3bab29b4c4dce908de0247c715423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130:39359), ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131:37775)
I20260807 03:37:29.857496 16606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4e0d750de56427e8a0af5bc030271c4" candidate_uuid: "9ca3bab29b4c4dce908de0247c715423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" is_pre_election: true
I20260807 03:37:29.858518 16606 raft_consensus.cc:2379] T a4e0d750de56427e8a0af5bc030271c4 P ec8c86cff51f48bf9ecc5177e6266b7b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9ca3bab29b4c4dce908de0247c715423 in current term 1: Already voted for candidate 05aa1d36d28a48138040f6c78c4b34a5 in this term.
I20260807 03:37:29.860328 16484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4e0d750de56427e8a0af5bc030271c4" candidate_uuid: "9ca3bab29b4c4dce908de0247c715423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05aa1d36d28a48138040f6c78c4b34a5" is_pre_election: true
I20260807 03:37:29.862121 16314 leader_election.cc:302] T a4e0d750de56427e8a0af5bc030271c4 P 9ca3bab29b4c4dce908de0247c715423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ca3bab29b4c4dce908de0247c715423; no voters: 05aa1d36d28a48138040f6c78c4b34a5, ec8c86cff51f48bf9ecc5177e6266b7b
I20260807 03:37:29.863487 16729 raft_consensus.cc:3037] T a4e0d750de56427e8a0af5bc030271c4 P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:37:29.873294 16729 raft_consensus.cc:2726] T a4e0d750de56427e8a0af5bc030271c4 P 9ca3bab29b4c4dce908de0247c715423 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:37:29.881238 16754 consensus_queue.cc:1035] T 5e409bda14f142959c8127f3bd2643fc P ec8c86cff51f48bf9ecc5177e6266b7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:37:29.887336 16696 raft_consensus.cc:482] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:37:29.887598 16365 raft_consensus.cc:3037] T d07d00a26d54438392d5db566a17627b P 9ca3bab29b4c4dce908de0247c715423 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:37:29.887804 16696 raft_consensus.cc:504] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:29.888742 16741 consensus_queue.cc:1035] T a4e0d750de56427e8a0af5bc030271c4 P 05aa1d36d28a48138040f6c78c4b34a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:37:29.890115 16696 leader_election.cc:288] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129:36375), 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130:39359)
I20260807 03:37:29.892720 16363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e0fd324ec84973b604e22e851d7b49" candidate_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ca3bab29b4c4dce908de0247c715423" is_pre_election: true
I20260807 03:37:29.893901 16363 raft_consensus.cc:2379] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ec8c86cff51f48bf9ecc5177e6266b7b in current term 1: Already voted for candidate 9ca3bab29b4c4dce908de0247c715423 in this term.
I20260807 03:37:29.896169 16479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e0fd324ec84973b604e22e851d7b49" candidate_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05aa1d36d28a48138040f6c78c4b34a5" is_pre_election: true
I20260807 03:37:29.897074 16479 raft_consensus.cc:2379] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ec8c86cff51f48bf9ecc5177e6266b7b in current term 1: Already voted for candidate 05aa1d36d28a48138040f6c78c4b34a5 in this term.
I20260807 03:37:29.898422 16557 leader_election.cc:302] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec8c86cff51f48bf9ecc5177e6266b7b; no voters: 05aa1d36d28a48138040f6c78c4b34a5, 9ca3bab29b4c4dce908de0247c715423
I20260807 03:37:29.899832 16754 raft_consensus.cc:3037] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:37:29.907155 16741 consensus_queue.cc:1035] T d07d00a26d54438392d5db566a17627b P 05aa1d36d28a48138040f6c78c4b34a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:37:29.911031 16754 raft_consensus.cc:2726] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:37:29.911438 16741 consensus_queue.cc:1035] T a4e0d750de56427e8a0af5bc030271c4 P 05aa1d36d28a48138040f6c78c4b34a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:37:30.024587 16754 consensus_queue.cc:1035] T 242a724d826340b693e5b0767a1ba28c P ec8c86cff51f48bf9ecc5177e6266b7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:37:30.035109 16750 consensus_queue.cc:1035] T 242a724d826340b693e5b0767a1ba28c P ec8c86cff51f48bf9ecc5177e6266b7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:37:30.136786 16786 consensus_queue.cc:1035] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:37:30.146574 16741 consensus_queue.cc:1035] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260807 03:37:30.512085 16279 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 03:37:30.852340 16786 raft_consensus.cc:482] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:37:30.852771 16786 raft_consensus.cc:504] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:30.854318 16786 leader_election.cc:288] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129:36375), ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131:37775)
I20260807 03:37:30.855051 16363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e0fd324ec84973b604e22e851d7b49" candidate_uuid: "05aa1d36d28a48138040f6c78c4b34a5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ca3bab29b4c4dce908de0247c715423" is_pre_election: true
I20260807 03:37:30.855324 16607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e0fd324ec84973b604e22e851d7b49" candidate_uuid: "05aa1d36d28a48138040f6c78c4b34a5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" is_pre_election: true
I20260807 03:37:30.855589 16363 raft_consensus.cc:2454] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05aa1d36d28a48138040f6c78c4b34a5 in term 1.
I20260807 03:37:30.855924 16607 raft_consensus.cc:2454] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05aa1d36d28a48138040f6c78c4b34a5 in term 1.
I20260807 03:37:30.856642 16432 leader_election.cc:302] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05aa1d36d28a48138040f6c78c4b34a5, 9ca3bab29b4c4dce908de0247c715423; no voters:
I20260807 03:37:30.857329 16786 raft_consensus.cc:2781] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:37:30.857652 16786 raft_consensus.cc:482] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:37:30.857889 16786 raft_consensus.cc:3037] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:37:30.862568 16786 raft_consensus.cc:504] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:30.864032 16786 leader_election.cc:288] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [CANDIDATE]: Term 2 election: Requested vote from peers 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129:36375), ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131:37775)
I20260807 03:37:30.864651 16363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e0fd324ec84973b604e22e851d7b49" candidate_uuid: "05aa1d36d28a48138040f6c78c4b34a5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ca3bab29b4c4dce908de0247c715423"
I20260807 03:37:30.865027 16363 raft_consensus.cc:3037] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:37:30.865005 16607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e0fd324ec84973b604e22e851d7b49" candidate_uuid: "05aa1d36d28a48138040f6c78c4b34a5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b"
I20260807 03:37:30.865582 16607 raft_consensus.cc:3037] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:37:30.869278 16363 raft_consensus.cc:2454] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05aa1d36d28a48138040f6c78c4b34a5 in term 2.
I20260807 03:37:30.869781 16607 raft_consensus.cc:2454] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05aa1d36d28a48138040f6c78c4b34a5 in term 2.
I20260807 03:37:30.870126 16432 leader_election.cc:302] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05aa1d36d28a48138040f6c78c4b34a5, 9ca3bab29b4c4dce908de0247c715423; no voters:
I20260807 03:37:30.870815 16786 raft_consensus.cc:2781] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:37:30.871192 16786 raft_consensus.cc:686] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 2 LEADER]: Becoming Leader. State: Replica: 05aa1d36d28a48138040f6c78c4b34a5, State: Running, Role: LEADER
I20260807 03:37:30.871892 16786 consensus_queue.cc:237] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:30.878334 16208 catalog_manager.cc:5495] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 reported cstate change: term changed from 0 to 2, leader changed from <none> to 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130). New cstate: current_term: 2 leader_uuid: "05aa1d36d28a48138040f6c78c4b34a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } health_report { overall_health: UNKNOWN } } }
I20260807 03:37:31.060622 16484 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f0191ce545e0490986331a9c562b056d"
dest_uuid: "05aa1d36d28a48138040f6c78c4b34a5"
mode: GRACEFUL
new_leader_uuid: "9ca3bab29b4c4dce908de0247c715423"
from {username='slave'} at 127.0.0.1:41690
I20260807 03:37:31.061242 16484 raft_consensus.cc:595] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 LEADER]: Received request to transfer leadership to TS 9ca3bab29b4c4dce908de0247c715423
I20260807 03:37:31.074313 16607 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fe184e38bf0d4e51bf42238d161ac738"
dest_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b"
mode: GRACEFUL
new_leader_uuid: "9ca3bab29b4c4dce908de0247c715423"
from {username='slave'} at 127.0.0.1:40168
I20260807 03:37:31.074793 16607 raft_consensus.cc:595] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b [term 1 LEADER]: Received request to transfer leadership to TS 9ca3bab29b4c4dce908de0247c715423
I20260807 03:37:31.078977 16484 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e9e0fd324ec84973b604e22e851d7b49"
dest_uuid: "05aa1d36d28a48138040f6c78c4b34a5"
mode: GRACEFUL
new_leader_uuid: "9ca3bab29b4c4dce908de0247c715423"
from {username='slave'} at 127.0.0.1:41690
I20260807 03:37:31.079461 16484 raft_consensus.cc:595] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 2 LEADER]: Received request to transfer leadership to TS 9ca3bab29b4c4dce908de0247c715423
I20260807 03:37:31.097220 16363 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "10452db9b82745b68c6c353b25eff5ef"
dest_uuid: "9ca3bab29b4c4dce908de0247c715423"
mode: GRACEFUL
new_leader_uuid: "05aa1d36d28a48138040f6c78c4b34a5"
from {username='slave'} at 127.0.0.1:49134
I20260807 03:37:31.097738 16363 raft_consensus.cc:595] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [term 1 LEADER]: Received request to transfer leadership to TS 05aa1d36d28a48138040f6c78c4b34a5
I20260807 03:37:31.102072 16607 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5e409bda14f142959c8127f3bd2643fc"
dest_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b"
mode: GRACEFUL
new_leader_uuid: "05aa1d36d28a48138040f6c78c4b34a5"
from {username='slave'} at 127.0.0.1:40168
I20260807 03:37:31.102545 16607 raft_consensus.cc:595] T 5e409bda14f142959c8127f3bd2643fc P ec8c86cff51f48bf9ecc5177e6266b7b [term 1 LEADER]: Received request to transfer leadership to TS 05aa1d36d28a48138040f6c78c4b34a5
I20260807 03:37:31.139565 16754 raft_consensus.cc:979] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b: : Instructing follower 9ca3bab29b4c4dce908de0247c715423 to start an election
I20260807 03:37:31.140172 16750 raft_consensus.cc:1067] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b [term 1 LEADER]: Signalling peer 9ca3bab29b4c4dce908de0247c715423 to start an election
I20260807 03:37:31.141547 16363 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fe184e38bf0d4e51bf42238d161ac738"
dest_uuid: "9ca3bab29b4c4dce908de0247c715423"
from {username='slave'} at 127.0.0.1:44380
I20260807 03:37:31.142082 16363 raft_consensus.cc:482] T fe184e38bf0d4e51bf42238d161ac738 P 9ca3bab29b4c4dce908de0247c715423 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:37:31.142364 16363 raft_consensus.cc:3037] T fe184e38bf0d4e51bf42238d161ac738 P 9ca3bab29b4c4dce908de0247c715423 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:37:31.147235 16363 raft_consensus.cc:504] T fe184e38bf0d4e51bf42238d161ac738 P 9ca3bab29b4c4dce908de0247c715423 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:31.148893 16363 leader_election.cc:288] T fe184e38bf0d4e51bf42238d161ac738 P 9ca3bab29b4c4dce908de0247c715423 [CANDIDATE]: Term 2 election: Requested vote from peers 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130:39359), ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131:37775)
I20260807 03:37:31.149600 16484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe184e38bf0d4e51bf42238d161ac738" candidate_uuid: "9ca3bab29b4c4dce908de0247c715423" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "05aa1d36d28a48138040f6c78c4b34a5"
I20260807 03:37:31.149799 16607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe184e38bf0d4e51bf42238d161ac738" candidate_uuid: "9ca3bab29b4c4dce908de0247c715423" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b"
I20260807 03:37:31.150250 16484 raft_consensus.cc:3037] T fe184e38bf0d4e51bf42238d161ac738 P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:37:31.150329 16607 raft_consensus.cc:3032] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:37:31.150836 16607 raft_consensus.cc:729] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ec8c86cff51f48bf9ecc5177e6266b7b, State: Running, Role: LEADER
I20260807 03:37:31.151468 16607 consensus_queue.cc:260] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:31.152570 16607 raft_consensus.cc:3037] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:37:31.155440 16484 raft_consensus.cc:2454] T fe184e38bf0d4e51bf42238d161ac738 P 05aa1d36d28a48138040f6c78c4b34a5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ca3bab29b4c4dce908de0247c715423 in term 2.
I20260807 03:37:31.156499 16314 leader_election.cc:302] T fe184e38bf0d4e51bf42238d161ac738 P 9ca3bab29b4c4dce908de0247c715423 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05aa1d36d28a48138040f6c78c4b34a5, 9ca3bab29b4c4dce908de0247c715423; no voters:
I20260807 03:37:31.157269 16733 raft_consensus.cc:2781] T fe184e38bf0d4e51bf42238d161ac738 P 9ca3bab29b4c4dce908de0247c715423 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:37:31.157917 16607 raft_consensus.cc:2454] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ca3bab29b4c4dce908de0247c715423 in term 2.
I20260807 03:37:31.158269 16733 raft_consensus.cc:686] T fe184e38bf0d4e51bf42238d161ac738 P 9ca3bab29b4c4dce908de0247c715423 [term 2 LEADER]: Becoming Leader. State: Replica: 9ca3bab29b4c4dce908de0247c715423, State: Running, Role: LEADER
I20260807 03:37:31.159092 16733 consensus_queue.cc:237] T fe184e38bf0d4e51bf42238d161ac738 P 9ca3bab29b4c4dce908de0247c715423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:31.165410 16208 catalog_manager.cc:5495] T fe184e38bf0d4e51bf42238d161ac738 P 9ca3bab29b4c4dce908de0247c715423 reported cstate change: term changed from 1 to 2, leader changed from ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131) to 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129). New cstate: current_term: 2 leader_uuid: "9ca3bab29b4c4dce908de0247c715423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } health_report { overall_health: UNKNOWN } } }
I20260807 03:37:31.182804 16819 raft_consensus.cc:979] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423: : Instructing follower 05aa1d36d28a48138040f6c78c4b34a5 to start an election
I20260807 03:37:31.183243 16733 raft_consensus.cc:1067] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [term 1 LEADER]: Signalling peer 05aa1d36d28a48138040f6c78c4b34a5 to start an election
I20260807 03:37:31.184477 16484 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "10452db9b82745b68c6c353b25eff5ef"
dest_uuid: "05aa1d36d28a48138040f6c78c4b34a5"
from {username='slave'} at 127.0.0.1:53828
I20260807 03:37:31.184912 16484 raft_consensus.cc:482] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:37:31.185202 16484 raft_consensus.cc:3037] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:37:31.189635 16484 raft_consensus.cc:504] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:31.191079 16484 leader_election.cc:288] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [CANDIDATE]: Term 2 election: Requested vote from peers 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129:36375), ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131:37775)
I20260807 03:37:31.191809 16363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10452db9b82745b68c6c353b25eff5ef" candidate_uuid: "05aa1d36d28a48138040f6c78c4b34a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9ca3bab29b4c4dce908de0247c715423"
I20260807 03:37:31.192103 16607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10452db9b82745b68c6c353b25eff5ef" candidate_uuid: "05aa1d36d28a48138040f6c78c4b34a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b"
I20260807 03:37:31.192497 16363 raft_consensus.cc:3032] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:37:31.192692 16607 raft_consensus.cc:3037] T 10452db9b82745b68c6c353b25eff5ef P ec8c86cff51f48bf9ecc5177e6266b7b [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:37:31.192826 16363 raft_consensus.cc:729] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9ca3bab29b4c4dce908de0247c715423, State: Running, Role: LEADER
I20260807 03:37:31.193591 16363 consensus_queue.cc:260] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:31.194569 16363 raft_consensus.cc:3037] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:37:31.197185 16607 raft_consensus.cc:2454] T 10452db9b82745b68c6c353b25eff5ef P ec8c86cff51f48bf9ecc5177e6266b7b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05aa1d36d28a48138040f6c78c4b34a5 in term 2.
I20260807 03:37:31.198029 16433 leader_election.cc:302] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05aa1d36d28a48138040f6c78c4b34a5, ec8c86cff51f48bf9ecc5177e6266b7b; no voters:
I20260807 03:37:31.198691 16786 raft_consensus.cc:2781] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:37:31.199158 16786 raft_consensus.cc:686] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [term 2 LEADER]: Becoming Leader. State: Replica: 05aa1d36d28a48138040f6c78c4b34a5, State: Running, Role: LEADER
I20260807 03:37:31.199337 16363 raft_consensus.cc:2454] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05aa1d36d28a48138040f6c78c4b34a5 in term 2.
I20260807 03:37:31.199913 16786 consensus_queue.cc:237] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:31.206594 16208 catalog_manager.cc:5495] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 reported cstate change: term changed from 1 to 2, leader changed from 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129) to 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130). New cstate: current_term: 2 leader_uuid: "05aa1d36d28a48138040f6c78c4b34a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } health_report { overall_health: UNKNOWN } } }
I20260807 03:37:31.332974 16754 raft_consensus.cc:979] T 5e409bda14f142959c8127f3bd2643fc P ec8c86cff51f48bf9ecc5177e6266b7b: : Instructing follower 05aa1d36d28a48138040f6c78c4b34a5 to start an election
I20260807 03:37:31.333374 16750 raft_consensus.cc:1067] T 5e409bda14f142959c8127f3bd2643fc P ec8c86cff51f48bf9ecc5177e6266b7b [term 1 LEADER]: Signalling peer 05aa1d36d28a48138040f6c78c4b34a5 to start an election
I20260807 03:37:31.334620 16484 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5e409bda14f142959c8127f3bd2643fc"
dest_uuid: "05aa1d36d28a48138040f6c78c4b34a5"
from {username='slave'} at 127.0.0.1:53832
I20260807 03:37:31.335107 16484 raft_consensus.cc:482] T 5e409bda14f142959c8127f3bd2643fc P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:37:31.335357 16484 raft_consensus.cc:3037] T 5e409bda14f142959c8127f3bd2643fc P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:37:31.339442 16484 raft_consensus.cc:504] T 5e409bda14f142959c8127f3bd2643fc P 05aa1d36d28a48138040f6c78c4b34a5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:31.340829 16484 leader_election.cc:288] T 5e409bda14f142959c8127f3bd2643fc P 05aa1d36d28a48138040f6c78c4b34a5 [CANDIDATE]: Term 2 election: Requested vote from peers 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129:36375), ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131:37775)
I20260807 03:37:31.341518 16363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e409bda14f142959c8127f3bd2643fc" candidate_uuid: "05aa1d36d28a48138040f6c78c4b34a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9ca3bab29b4c4dce908de0247c715423"
I20260807 03:37:31.341804 16607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e409bda14f142959c8127f3bd2643fc" candidate_uuid: "05aa1d36d28a48138040f6c78c4b34a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b"
I20260807 03:37:31.342085 16363 raft_consensus.cc:3037] T 5e409bda14f142959c8127f3bd2643fc P 9ca3bab29b4c4dce908de0247c715423 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:37:31.342372 16607 raft_consensus.cc:3032] T 5e409bda14f142959c8127f3bd2643fc P ec8c86cff51f48bf9ecc5177e6266b7b [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:37:31.342654 16607 raft_consensus.cc:729] T 5e409bda14f142959c8127f3bd2643fc P ec8c86cff51f48bf9ecc5177e6266b7b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ec8c86cff51f48bf9ecc5177e6266b7b, State: Running, Role: LEADER
I20260807 03:37:31.343268 16607 consensus_queue.cc:260] T 5e409bda14f142959c8127f3bd2643fc P ec8c86cff51f48bf9ecc5177e6266b7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:31.344388 16607 raft_consensus.cc:3037] T 5e409bda14f142959c8127f3bd2643fc P ec8c86cff51f48bf9ecc5177e6266b7b [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:37:31.346480 16363 raft_consensus.cc:2454] T 5e409bda14f142959c8127f3bd2643fc P 9ca3bab29b4c4dce908de0247c715423 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05aa1d36d28a48138040f6c78c4b34a5 in term 2.
I20260807 03:37:31.347316 16432 leader_election.cc:302] T 5e409bda14f142959c8127f3bd2643fc P 05aa1d36d28a48138040f6c78c4b34a5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05aa1d36d28a48138040f6c78c4b34a5, 9ca3bab29b4c4dce908de0247c715423; no voters:
I20260807 03:37:31.347944 16786 raft_consensus.cc:2781] T 5e409bda14f142959c8127f3bd2643fc P 05aa1d36d28a48138040f6c78c4b34a5 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:37:31.348407 16786 raft_consensus.cc:686] T 5e409bda14f142959c8127f3bd2643fc P 05aa1d36d28a48138040f6c78c4b34a5 [term 2 LEADER]: Becoming Leader. State: Replica: 05aa1d36d28a48138040f6c78c4b34a5, State: Running, Role: LEADER
I20260807 03:37:31.349112 16786 consensus_queue.cc:237] T 5e409bda14f142959c8127f3bd2643fc P 05aa1d36d28a48138040f6c78c4b34a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:31.349984 16607 raft_consensus.cc:2454] T 5e409bda14f142959c8127f3bd2643fc P ec8c86cff51f48bf9ecc5177e6266b7b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05aa1d36d28a48138040f6c78c4b34a5 in term 2.
I20260807 03:37:31.353818 16363 raft_consensus.cc:1261] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [term 2 FOLLOWER]: Refusing update from remote peer 05aa1d36d28a48138040f6c78c4b34a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260807 03:37:31.354863 16786 consensus_queue.cc:1035] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:37:31.356541 16208 catalog_manager.cc:5495] T 5e409bda14f142959c8127f3bd2643fc P 05aa1d36d28a48138040f6c78c4b34a5 reported cstate change: term changed from 1 to 2, leader changed from ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131) to 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130). New cstate: current_term: 2 leader_uuid: "05aa1d36d28a48138040f6c78c4b34a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } health_report { overall_health: UNKNOWN } } }
I20260807 03:37:31.360802 16741 raft_consensus.cc:979] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5: : Instructing follower 9ca3bab29b4c4dce908de0247c715423 to start an election
I20260807 03:37:31.361467 16786 raft_consensus.cc:1067] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 2 LEADER]: Signalling peer 9ca3bab29b4c4dce908de0247c715423 to start an election
I20260807 03:37:31.366235 16363 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e9e0fd324ec84973b604e22e851d7b49"
dest_uuid: "9ca3bab29b4c4dce908de0247c715423"
from {username='slave'} at 127.0.0.1:44368
I20260807 03:37:31.366674 16363 raft_consensus.cc:482] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:37:31.366983 16363 raft_consensus.cc:3037] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:37:31.372159 16606 raft_consensus.cc:1261] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b [term 2 FOLLOWER]: Refusing update from remote peer 05aa1d36d28a48138040f6c78c4b34a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260807 03:37:31.373344 16741 consensus_queue.cc:1035] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:37:31.374725 16363 raft_consensus.cc:504] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:31.376479 16363 leader_election.cc:288] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [CANDIDATE]: Term 3 election: Requested vote from peers 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130:39359), ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131:37775)
I20260807 03:37:31.377487 16484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e0fd324ec84973b604e22e851d7b49" candidate_uuid: "9ca3bab29b4c4dce908de0247c715423" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "05aa1d36d28a48138040f6c78c4b34a5"
I20260807 03:37:31.377837 16607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e0fd324ec84973b604e22e851d7b49" candidate_uuid: "9ca3bab29b4c4dce908de0247c715423" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b"
I20260807 03:37:31.378032 16484 raft_consensus.cc:3032] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 03:37:31.378320 16484 raft_consensus.cc:729] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 05aa1d36d28a48138040f6c78c4b34a5, State: Running, Role: LEADER
I20260807 03:37:31.379051 16484 consensus_queue.cc:260] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:31.379933 16484 raft_consensus.cc:3037] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:37:31.384940 16484 raft_consensus.cc:2454] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ca3bab29b4c4dce908de0247c715423 in term 3.
I20260807 03:37:31.385798 16314 leader_election.cc:302] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05aa1d36d28a48138040f6c78c4b34a5, 9ca3bab29b4c4dce908de0247c715423; no voters: ec8c86cff51f48bf9ecc5177e6266b7b
I20260807 03:37:31.386495 16733 raft_consensus.cc:2781] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:37:31.386979 16733 raft_consensus.cc:686] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [term 3 LEADER]: Becoming Leader. State: Replica: 9ca3bab29b4c4dce908de0247c715423, State: Running, Role: LEADER
I20260807 03:37:31.387670 16733 consensus_queue.cc:237] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:31.393579 16207 catalog_manager.cc:5495] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 reported cstate change: term changed from 2 to 3, leader changed from 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130) to 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129). New cstate: current_term: 3 leader_uuid: "9ca3bab29b4c4dce908de0247c715423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } health_report { overall_health: UNKNOWN } } }
I20260807 03:37:31.438309 16829 raft_consensus.cc:979] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5: : Instructing follower 9ca3bab29b4c4dce908de0247c715423 to start an election
I20260807 03:37:31.438719 16741 raft_consensus.cc:1067] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 LEADER]: Signalling peer 9ca3bab29b4c4dce908de0247c715423 to start an election
I20260807 03:37:31.440017 16363 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f0191ce545e0490986331a9c562b056d"
dest_uuid: "9ca3bab29b4c4dce908de0247c715423"
from {username='slave'} at 127.0.0.1:44368
I20260807 03:37:31.440518 16363 raft_consensus.cc:482] T f0191ce545e0490986331a9c562b056d P 9ca3bab29b4c4dce908de0247c715423 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:37:31.440858 16363 raft_consensus.cc:3037] T f0191ce545e0490986331a9c562b056d P 9ca3bab29b4c4dce908de0247c715423 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:37:31.446273 16363 raft_consensus.cc:504] T f0191ce545e0490986331a9c562b056d P 9ca3bab29b4c4dce908de0247c715423 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:31.447679 16363 leader_election.cc:288] T f0191ce545e0490986331a9c562b056d P 9ca3bab29b4c4dce908de0247c715423 [CANDIDATE]: Term 2 election: Requested vote from peers 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130:39359), ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131:37775)
I20260807 03:37:31.448498 16484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0191ce545e0490986331a9c562b056d" candidate_uuid: "9ca3bab29b4c4dce908de0247c715423" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "05aa1d36d28a48138040f6c78c4b34a5"
I20260807 03:37:31.448715 16606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0191ce545e0490986331a9c562b056d" candidate_uuid: "9ca3bab29b4c4dce908de0247c715423" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b"
I20260807 03:37:31.449090 16484 raft_consensus.cc:3032] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:37:31.449200 16606 raft_consensus.cc:3037] T f0191ce545e0490986331a9c562b056d P ec8c86cff51f48bf9ecc5177e6266b7b [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:37:31.449437 16484 raft_consensus.cc:729] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 05aa1d36d28a48138040f6c78c4b34a5, State: Running, Role: LEADER
I20260807 03:37:31.450184 16484 consensus_queue.cc:260] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:31.451368 16484 raft_consensus.cc:3037] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:37:31.455318 16606 raft_consensus.cc:2454] T f0191ce545e0490986331a9c562b056d P ec8c86cff51f48bf9ecc5177e6266b7b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ca3bab29b4c4dce908de0247c715423 in term 2.
I20260807 03:37:31.456339 16314 leader_election.cc:302] T f0191ce545e0490986331a9c562b056d P 9ca3bab29b4c4dce908de0247c715423 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ca3bab29b4c4dce908de0247c715423, ec8c86cff51f48bf9ecc5177e6266b7b; no voters:
I20260807 03:37:31.457000 16733 raft_consensus.cc:2781] T f0191ce545e0490986331a9c562b056d P 9ca3bab29b4c4dce908de0247c715423 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:37:31.457172 16484 raft_consensus.cc:2454] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ca3bab29b4c4dce908de0247c715423 in term 2.
I20260807 03:37:31.457396 16733 raft_consensus.cc:686] T f0191ce545e0490986331a9c562b056d P 9ca3bab29b4c4dce908de0247c715423 [term 2 LEADER]: Becoming Leader. State: Replica: 9ca3bab29b4c4dce908de0247c715423, State: Running, Role: LEADER
I20260807 03:37:31.458185 16733 consensus_queue.cc:237] T f0191ce545e0490986331a9c562b056d P 9ca3bab29b4c4dce908de0247c715423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:31.464725 16208 catalog_manager.cc:5495] T f0191ce545e0490986331a9c562b056d P 9ca3bab29b4c4dce908de0247c715423 reported cstate change: term changed from 1 to 2, leader changed from 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130) to 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129). New cstate: current_term: 2 leader_uuid: "9ca3bab29b4c4dce908de0247c715423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } health_report { overall_health: UNKNOWN } } }
I20260807 03:37:31.549276 16606 raft_consensus.cc:1261] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b [term 2 FOLLOWER]: Refusing update from remote peer 9ca3bab29b4c4dce908de0247c715423: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:37:31.550377 16733 consensus_queue.cc:1035] T fe184e38bf0d4e51bf42238d161ac738 P 9ca3bab29b4c4dce908de0247c715423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:37:31.558923 16484 raft_consensus.cc:1261] T fe184e38bf0d4e51bf42238d161ac738 P 05aa1d36d28a48138040f6c78c4b34a5 [term 2 FOLLOWER]: Refusing update from remote peer 9ca3bab29b4c4dce908de0247c715423: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:37:31.561038 16844 consensus_queue.cc:1035] T fe184e38bf0d4e51bf42238d161ac738 P 9ca3bab29b4c4dce908de0247c715423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 03:37:31.622483 16606 raft_consensus.cc:1261] T 10452db9b82745b68c6c353b25eff5ef P ec8c86cff51f48bf9ecc5177e6266b7b [term 2 FOLLOWER]: Refusing update from remote peer 05aa1d36d28a48138040f6c78c4b34a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:37:31.623553 16741 consensus_queue.cc:1035] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:37:31.631003 16363 raft_consensus.cc:1261] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [term 2 FOLLOWER]: Refusing update from remote peer 05aa1d36d28a48138040f6c78c4b34a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:37:31.632386 16741 consensus_queue.cc:1035] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:37:31.755975 16363 raft_consensus.cc:1261] T 5e409bda14f142959c8127f3bd2643fc P 9ca3bab29b4c4dce908de0247c715423 [term 2 FOLLOWER]: Refusing update from remote peer 05aa1d36d28a48138040f6c78c4b34a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:37:31.757112 16741 consensus_queue.cc:1035] T 5e409bda14f142959c8127f3bd2643fc P 05aa1d36d28a48138040f6c78c4b34a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:37:31.764330 16606 raft_consensus.cc:1261] T 5e409bda14f142959c8127f3bd2643fc P ec8c86cff51f48bf9ecc5177e6266b7b [term 2 FOLLOWER]: Refusing update from remote peer 05aa1d36d28a48138040f6c78c4b34a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:37:31.765681 16741 consensus_queue.cc:1035] T 5e409bda14f142959c8127f3bd2643fc P 05aa1d36d28a48138040f6c78c4b34a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:37:31.768867 16484 raft_consensus.cc:1261] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 3 FOLLOWER]: Refusing update from remote peer 9ca3bab29b4c4dce908de0247c715423: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:37:31.770386 16844 consensus_queue.cc:1035] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:37:31.778546 16606 raft_consensus.cc:3037] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:37:31.786307 16606 raft_consensus.cc:1261] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b [term 3 FOLLOWER]: Refusing update from remote peer 9ca3bab29b4c4dce908de0247c715423: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:37:31.787801 16819 consensus_queue.cc:1035] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:37:31.857283 16479 raft_consensus.cc:1261] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5 [term 2 FOLLOWER]: Refusing update from remote peer 9ca3bab29b4c4dce908de0247c715423: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:37:31.858382 16819 consensus_queue.cc:1035] T f0191ce545e0490986331a9c562b056d P 9ca3bab29b4c4dce908de0247c715423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:37:31.867027 16606 raft_consensus.cc:1261] T f0191ce545e0490986331a9c562b056d P ec8c86cff51f48bf9ecc5177e6266b7b [term 2 FOLLOWER]: Refusing update from remote peer 9ca3bab29b4c4dce908de0247c715423: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:37:31.869362 16855 consensus_queue.cc:1035] T f0191ce545e0490986331a9c562b056d P 9ca3bab29b4c4dce908de0247c715423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:37:34.133291 16154 auto_leader_rebalancer-test.cc:269] The leader distribution is
I20260807 03:37:34.134212 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:37:34.155944 16479 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "10452db9b82745b68c6c353b25eff5ef"
dest_uuid: "05aa1d36d28a48138040f6c78c4b34a5"
mode: GRACEFUL
new_leader_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b"
from {username='slave'} at 127.0.0.1:41698
I20260807 03:37:34.156507 16479 raft_consensus.cc:595] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [term 2 LEADER]: Received request to transfer leadership to TS ec8c86cff51f48bf9ecc5177e6266b7b
I20260807 03:37:34.168030 16363 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e9e0fd324ec84973b604e22e851d7b49"
dest_uuid: "9ca3bab29b4c4dce908de0247c715423"
mode: GRACEFUL
new_leader_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b"
from {username='slave'} at 127.0.0.1:49150
I20260807 03:37:34.168584 16363 raft_consensus.cc:595] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [term 3 LEADER]: Received request to transfer leadership to TS ec8c86cff51f48bf9ecc5177e6266b7b
I20260807 03:37:34.169600 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260807 03:37:34.170045 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:37:34.399303 16871 raft_consensus.cc:979] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423: : Instructing follower ec8c86cff51f48bf9ecc5177e6266b7b to start an election
I20260807 03:37:34.399739 16861 raft_consensus.cc:1067] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [term 3 LEADER]: Signalling peer ec8c86cff51f48bf9ecc5177e6266b7b to start an election
I20260807 03:37:34.400995 16606 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e9e0fd324ec84973b604e22e851d7b49"
dest_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b"
from {username='slave'} at 127.0.0.1:45592
I20260807 03:37:34.401403 16606 raft_consensus.cc:482] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:37:34.401636 16606 raft_consensus.cc:3037] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:37:34.406224 16606 raft_consensus.cc:504] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:34.407884 16606 leader_election.cc:288] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b [CANDIDATE]: Term 4 election: Requested vote from peers 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129:36375), 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130:39359)
I20260807 03:37:34.408567 16363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e0fd324ec84973b604e22e851d7b49" candidate_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "9ca3bab29b4c4dce908de0247c715423"
I20260807 03:37:34.409143 16363 raft_consensus.cc:3032] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:37:34.409530 16363 raft_consensus.cc:729] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9ca3bab29b4c4dce908de0247c715423, State: Running, Role: LEADER
I20260807 03:37:34.409165 16479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e0fd324ec84973b604e22e851d7b49" candidate_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "05aa1d36d28a48138040f6c78c4b34a5"
I20260807 03:37:34.410207 16479 raft_consensus.cc:3037] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:37:34.410331 16363 consensus_queue.cc:260] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:34.411533 16363 raft_consensus.cc:3037] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:37:34.415438 16479 raft_consensus.cc:2454] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec8c86cff51f48bf9ecc5177e6266b7b in term 4.
I20260807 03:37:34.416138 16363 raft_consensus.cc:2454] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec8c86cff51f48bf9ecc5177e6266b7b in term 4.
I20260807 03:37:34.416457 16557 leader_election.cc:302] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05aa1d36d28a48138040f6c78c4b34a5, ec8c86cff51f48bf9ecc5177e6266b7b; no voters:
I20260807 03:37:34.417232 16754 raft_consensus.cc:2781] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b [term 4 FOLLOWER]: Leader election won for term 4
I20260807 03:37:34.418257 16754 raft_consensus.cc:686] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b [term 4 LEADER]: Becoming Leader. State: Replica: ec8c86cff51f48bf9ecc5177e6266b7b, State: Running, Role: LEADER
I20260807 03:37:34.419035 16754 consensus_queue.cc:237] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:34.425364 16207 catalog_manager.cc:5495] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b reported cstate change: term changed from 3 to 4, leader changed from 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129) to ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131). New cstate: current_term: 4 leader_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } health_report { overall_health: HEALTHY } } }
I20260807 03:37:34.673368 16741 raft_consensus.cc:979] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5: : Instructing follower ec8c86cff51f48bf9ecc5177e6266b7b to start an election
I20260807 03:37:34.673784 16881 raft_consensus.cc:1067] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [term 2 LEADER]: Signalling peer ec8c86cff51f48bf9ecc5177e6266b7b to start an election
I20260807 03:37:34.675117 16604 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "10452db9b82745b68c6c353b25eff5ef"
dest_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b"
from {username='slave'} at 127.0.0.1:45596
I20260807 03:37:34.675632 16604 raft_consensus.cc:482] T 10452db9b82745b68c6c353b25eff5ef P ec8c86cff51f48bf9ecc5177e6266b7b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:37:34.675941 16604 raft_consensus.cc:3037] T 10452db9b82745b68c6c353b25eff5ef P ec8c86cff51f48bf9ecc5177e6266b7b [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:37:34.681528 16604 raft_consensus.cc:504] T 10452db9b82745b68c6c353b25eff5ef P ec8c86cff51f48bf9ecc5177e6266b7b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:34.683578 16604 leader_election.cc:288] T 10452db9b82745b68c6c353b25eff5ef P ec8c86cff51f48bf9ecc5177e6266b7b [CANDIDATE]: Term 3 election: Requested vote from peers 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129:36375), 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130:39359)
I20260807 03:37:34.684083 16363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10452db9b82745b68c6c353b25eff5ef" candidate_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9ca3bab29b4c4dce908de0247c715423"
I20260807 03:37:34.684195 16479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10452db9b82745b68c6c353b25eff5ef" candidate_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "05aa1d36d28a48138040f6c78c4b34a5"
I20260807 03:37:34.684837 16363 raft_consensus.cc:3037] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:37:34.684864 16479 raft_consensus.cc:3032] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 03:37:34.685310 16479 raft_consensus.cc:729] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 05aa1d36d28a48138040f6c78c4b34a5, State: Running, Role: LEADER
I20260807 03:37:34.686012 16479 consensus_queue.cc:260] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:34.687042 16479 raft_consensus.cc:3037] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:37:34.689471 16363 raft_consensus.cc:2454] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec8c86cff51f48bf9ecc5177e6266b7b in term 3.
I20260807 03:37:34.690327 16556 leader_election.cc:302] T 10452db9b82745b68c6c353b25eff5ef P ec8c86cff51f48bf9ecc5177e6266b7b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ca3bab29b4c4dce908de0247c715423, ec8c86cff51f48bf9ecc5177e6266b7b; no voters:
I20260807 03:37:34.690963 16754 raft_consensus.cc:2781] T 10452db9b82745b68c6c353b25eff5ef P ec8c86cff51f48bf9ecc5177e6266b7b [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:37:34.691331 16754 raft_consensus.cc:686] T 10452db9b82745b68c6c353b25eff5ef P ec8c86cff51f48bf9ecc5177e6266b7b [term 3 LEADER]: Becoming Leader. State: Replica: ec8c86cff51f48bf9ecc5177e6266b7b, State: Running, Role: LEADER
I20260807 03:37:34.692127 16479 raft_consensus.cc:2454] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec8c86cff51f48bf9ecc5177e6266b7b in term 3.
I20260807 03:37:34.692044 16754 consensus_queue.cc:237] T 10452db9b82745b68c6c353b25eff5ef P ec8c86cff51f48bf9ecc5177e6266b7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:34.698873 16207 catalog_manager.cc:5495] T 10452db9b82745b68c6c353b25eff5ef P ec8c86cff51f48bf9ecc5177e6266b7b reported cstate change: term changed from 2 to 3, leader changed from 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130) to ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131). New cstate: current_term: 3 leader_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } health_report { overall_health: HEALTHY } } }
I20260807 03:37:34.814025 16479 raft_consensus.cc:1261] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 4 FOLLOWER]: Refusing update from remote peer ec8c86cff51f48bf9ecc5177e6266b7b: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 03:37:34.815358 16754 consensus_queue.cc:1035] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:37:34.824307 16363 raft_consensus.cc:1261] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [term 4 FOLLOWER]: Refusing update from remote peer ec8c86cff51f48bf9ecc5177e6266b7b: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 03:37:34.825987 16754 consensus_queue.cc:1035] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:37:35.078331 16363 raft_consensus.cc:1261] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [term 3 FOLLOWER]: Refusing update from remote peer ec8c86cff51f48bf9ecc5177e6266b7b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 03:37:35.079505 16754 consensus_queue.cc:1035] T 10452db9b82745b68c6c353b25eff5ef P ec8c86cff51f48bf9ecc5177e6266b7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:37:35.087378 16479 raft_consensus.cc:1261] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [term 3 FOLLOWER]: Refusing update from remote peer ec8c86cff51f48bf9ecc5177e6266b7b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 03:37:35.088944 16872 consensus_queue.cc:1035] T 10452db9b82745b68c6c353b25eff5ef P ec8c86cff51f48bf9ecc5177e6266b7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:37:35.170912 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:37:35.177765 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 03:37:35.178088 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:37:36.179143 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:37:36.184489 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 03:37:36.184871 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:37:37.185762 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:37:37.191001 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 03:37:37.191413 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:37:38.192147 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:37:38.197398 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 03:37:38.197788 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:37:39.198702 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:37:39.203800 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 03:37:39.204213 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:37:40.204962 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:37:40.209653 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 03:37:40.210028 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:37:41.210860 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:37:41.215723 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 03:37:41.216111 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:37:42.216887 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:37:42.221455 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 03:37:42.221860 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:37:43.222582 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:37:43.227334 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 03:37:43.227669 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:37:44.251798 16154 auto_leader_rebalancer-test.cc:288] The leader distribution is
I20260807 03:37:44.255710 16365 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f0191ce545e0490986331a9c562b056d"
dest_uuid: "9ca3bab29b4c4dce908de0247c715423"
mode: GRACEFUL
new_leader_uuid: "05aa1d36d28a48138040f6c78c4b34a5"
from {username='slave'} at 127.0.0.1:49134
I20260807 03:37:44.256680 16365 raft_consensus.cc:595] T f0191ce545e0490986331a9c562b056d P 9ca3bab29b4c4dce908de0247c715423 [term 2 LEADER]: Received request to transfer leadership to TS 05aa1d36d28a48138040f6c78c4b34a5
I20260807 03:37:44.261091 16365 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fe184e38bf0d4e51bf42238d161ac738"
dest_uuid: "9ca3bab29b4c4dce908de0247c715423"
mode: GRACEFUL
new_leader_uuid: "05aa1d36d28a48138040f6c78c4b34a5"
from {username='slave'} at 127.0.0.1:49134
I20260807 03:37:44.261518 16365 raft_consensus.cc:595] T fe184e38bf0d4e51bf42238d161ac738 P 9ca3bab29b4c4dce908de0247c715423 [term 2 LEADER]: Received request to transfer leadership to TS 05aa1d36d28a48138040f6c78c4b34a5
I20260807 03:37:44.265193 16604 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e9e0fd324ec84973b604e22e851d7b49"
dest_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b"
mode: GRACEFUL
new_leader_uuid: "05aa1d36d28a48138040f6c78c4b34a5"
from {username='slave'} at 127.0.0.1:40168
I20260807 03:37:44.265591 16604 raft_consensus.cc:595] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b [term 4 LEADER]: Received request to transfer leadership to TS 05aa1d36d28a48138040f6c78c4b34a5
I20260807 03:37:44.269524 16365 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "485e8d9a337d4e4f9207f737e3a8812b"
dest_uuid: "9ca3bab29b4c4dce908de0247c715423"
mode: GRACEFUL
new_leader_uuid: "05aa1d36d28a48138040f6c78c4b34a5"
from {username='slave'} at 127.0.0.1:49134
I20260807 03:37:44.270025 16365 raft_consensus.cc:595] T 485e8d9a337d4e4f9207f737e3a8812b P 9ca3bab29b4c4dce908de0247c715423 [term 1 LEADER]: Received request to transfer leadership to TS 05aa1d36d28a48138040f6c78c4b34a5
I20260807 03:37:44.276062 16604 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "10452db9b82745b68c6c353b25eff5ef"
dest_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b"
mode: GRACEFUL
new_leader_uuid: "05aa1d36d28a48138040f6c78c4b34a5"
from {username='slave'} at 127.0.0.1:40168
I20260807 03:37:44.276542 16604 raft_consensus.cc:595] T 10452db9b82745b68c6c353b25eff5ef P ec8c86cff51f48bf9ecc5177e6266b7b [term 3 LEADER]: Received request to transfer leadership to TS 05aa1d36d28a48138040f6c78c4b34a5
I20260807 03:37:44.381035 16904 raft_consensus.cc:979] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b: : Instructing follower 05aa1d36d28a48138040f6c78c4b34a5 to start an election
I20260807 03:37:44.382304 16872 raft_consensus.cc:979] T 10452db9b82745b68c6c353b25eff5ef P ec8c86cff51f48bf9ecc5177e6266b7b: : Instructing follower 05aa1d36d28a48138040f6c78c4b34a5 to start an election
I20260807 03:37:44.382488 16906 raft_consensus.cc:1067] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b [term 4 LEADER]: Signalling peer 05aa1d36d28a48138040f6c78c4b34a5 to start an election
I20260807 03:37:44.382879 16872 raft_consensus.cc:1067] T 10452db9b82745b68c6c353b25eff5ef P ec8c86cff51f48bf9ecc5177e6266b7b [term 3 LEADER]: Signalling peer 05aa1d36d28a48138040f6c78c4b34a5 to start an election
I20260807 03:37:44.384516 16479 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e9e0fd324ec84973b604e22e851d7b49"
dest_uuid: "05aa1d36d28a48138040f6c78c4b34a5"
from {username='slave'} at 127.0.0.1:53832
I20260807 03:37:44.385037 16479 raft_consensus.cc:482] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:37:44.385084 16484 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "10452db9b82745b68c6c353b25eff5ef"
dest_uuid: "05aa1d36d28a48138040f6c78c4b34a5"
from {username='slave'} at 127.0.0.1:53832
I20260807 03:37:44.385568 16479 raft_consensus.cc:3037] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:37:44.385702 16484 raft_consensus.cc:482] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:37:44.386125 16484 raft_consensus.cc:3037] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:37:44.393533 16479 raft_consensus.cc:504] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:44.393482 16484 raft_consensus.cc:504] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:44.396924 16484 leader_election.cc:288] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [CANDIDATE]: Term 4 election: Requested vote from peers 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129:36375), ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131:37775)
I20260807 03:37:44.398053 16604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e0fd324ec84973b604e22e851d7b49" candidate_uuid: "05aa1d36d28a48138040f6c78c4b34a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b"
I20260807 03:37:44.398267 16364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10452db9b82745b68c6c353b25eff5ef" candidate_uuid: "05aa1d36d28a48138040f6c78c4b34a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "9ca3bab29b4c4dce908de0247c715423"
I20260807 03:37:44.398053 16365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e0fd324ec84973b604e22e851d7b49" candidate_uuid: "05aa1d36d28a48138040f6c78c4b34a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "9ca3bab29b4c4dce908de0247c715423"
I20260807 03:37:44.397619 16479 leader_election.cc:288] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [CANDIDATE]: Term 5 election: Requested vote from peers 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129:36375), ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131:37775)
I20260807 03:37:44.398815 16606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10452db9b82745b68c6c353b25eff5ef" candidate_uuid: "05aa1d36d28a48138040f6c78c4b34a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b"
I20260807 03:37:44.399111 16364 raft_consensus.cc:3037] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:37:44.399169 16365 raft_consensus.cc:3037] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:37:44.399518 16606 raft_consensus.cc:3032] T 10452db9b82745b68c6c353b25eff5ef P ec8c86cff51f48bf9ecc5177e6266b7b [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:37:44.399791 16604 raft_consensus.cc:3032] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b [term 4 LEADER]: Stepping down as leader of term 4
I20260807 03:37:44.399950 16606 raft_consensus.cc:729] T 10452db9b82745b68c6c353b25eff5ef P ec8c86cff51f48bf9ecc5177e6266b7b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ec8c86cff51f48bf9ecc5177e6266b7b, State: Running, Role: LEADER
I20260807 03:37:44.400254 16604 raft_consensus.cc:729] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ec8c86cff51f48bf9ecc5177e6266b7b, State: Running, Role: LEADER
I20260807 03:37:44.400887 16606 consensus_queue.cc:260] T 10452db9b82745b68c6c353b25eff5ef P ec8c86cff51f48bf9ecc5177e6266b7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:44.401149 16604 consensus_queue.cc:260] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:44.402087 16606 raft_consensus.cc:3037] T 10452db9b82745b68c6c353b25eff5ef P ec8c86cff51f48bf9ecc5177e6266b7b [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:37:44.402338 16604 raft_consensus.cc:3037] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:37:44.406791 16365 raft_consensus.cc:2454] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05aa1d36d28a48138040f6c78c4b34a5 in term 5.
I20260807 03:37:44.407733 16364 raft_consensus.cc:2454] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05aa1d36d28a48138040f6c78c4b34a5 in term 4.
I20260807 03:37:44.408735 16432 leader_election.cc:302] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05aa1d36d28a48138040f6c78c4b34a5, 9ca3bab29b4c4dce908de0247c715423; no voters:
I20260807 03:37:44.409536 16881 raft_consensus.cc:2781] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 03:37:44.410153 16606 raft_consensus.cc:2454] T 10452db9b82745b68c6c353b25eff5ef P ec8c86cff51f48bf9ecc5177e6266b7b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05aa1d36d28a48138040f6c78c4b34a5 in term 4.
I20260807 03:37:44.410239 16432 leader_election.cc:302] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05aa1d36d28a48138040f6c78c4b34a5, 9ca3bab29b4c4dce908de0247c715423; no voters:
I20260807 03:37:44.410800 16881 raft_consensus.cc:686] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 5 LEADER]: Becoming Leader. State: Replica: 05aa1d36d28a48138040f6c78c4b34a5, State: Running, Role: LEADER
I20260807 03:37:44.409564 16604 raft_consensus.cc:2454] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05aa1d36d28a48138040f6c78c4b34a5 in term 5.
I20260807 03:37:44.411764 16881 consensus_queue.cc:237] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:44.412365 16908 raft_consensus.cc:2781] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 03:37:44.413079 16908 raft_consensus.cc:686] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [term 4 LEADER]: Becoming Leader. State: Replica: 05aa1d36d28a48138040f6c78c4b34a5, State: Running, Role: LEADER
I20260807 03:37:44.413862 16908 consensus_queue.cc:237] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:44.420521 16207 catalog_manager.cc:5495] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 reported cstate change: term changed from 4 to 5, leader changed from ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131) to 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130). New cstate: current_term: 5 leader_uuid: "05aa1d36d28a48138040f6c78c4b34a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } health_report { overall_health: UNKNOWN } } }
I20260807 03:37:44.435137 16207 catalog_manager.cc:5495] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 reported cstate change: term changed from 3 to 4, leader changed from ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131) to 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130). New cstate: current_term: 4 leader_uuid: "05aa1d36d28a48138040f6c78c4b34a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } health_report { overall_health: UNKNOWN } } }
I20260807 03:37:44.503266 16861 raft_consensus.cc:979] T 485e8d9a337d4e4f9207f737e3a8812b P 9ca3bab29b4c4dce908de0247c715423: : Instructing follower 05aa1d36d28a48138040f6c78c4b34a5 to start an election
I20260807 03:37:44.503784 16924 raft_consensus.cc:1067] T 485e8d9a337d4e4f9207f737e3a8812b P 9ca3bab29b4c4dce908de0247c715423 [term 1 LEADER]: Signalling peer 05aa1d36d28a48138040f6c78c4b34a5 to start an election
I20260807 03:37:44.505290 16479 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "485e8d9a337d4e4f9207f737e3a8812b"
dest_uuid: "05aa1d36d28a48138040f6c78c4b34a5"
from {username='slave'} at 127.0.0.1:53828
I20260807 03:37:44.505779 16479 raft_consensus.cc:482] T 485e8d9a337d4e4f9207f737e3a8812b P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:37:44.506047 16479 raft_consensus.cc:3037] T 485e8d9a337d4e4f9207f737e3a8812b P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:37:44.510016 16479 raft_consensus.cc:504] T 485e8d9a337d4e4f9207f737e3a8812b P 05aa1d36d28a48138040f6c78c4b34a5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:44.511431 16479 leader_election.cc:288] T 485e8d9a337d4e4f9207f737e3a8812b P 05aa1d36d28a48138040f6c78c4b34a5 [CANDIDATE]: Term 2 election: Requested vote from peers 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129:36375), ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131:37775)
I20260807 03:37:44.512308 16364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "485e8d9a337d4e4f9207f737e3a8812b" candidate_uuid: "05aa1d36d28a48138040f6c78c4b34a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9ca3bab29b4c4dce908de0247c715423"
I20260807 03:37:44.512557 16604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "485e8d9a337d4e4f9207f737e3a8812b" candidate_uuid: "05aa1d36d28a48138040f6c78c4b34a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b"
I20260807 03:37:44.512879 16364 raft_consensus.cc:3032] T 485e8d9a337d4e4f9207f737e3a8812b P 9ca3bab29b4c4dce908de0247c715423 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:37:44.513084 16604 raft_consensus.cc:3037] T 485e8d9a337d4e4f9207f737e3a8812b P ec8c86cff51f48bf9ecc5177e6266b7b [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:37:44.513202 16364 raft_consensus.cc:729] T 485e8d9a337d4e4f9207f737e3a8812b P 9ca3bab29b4c4dce908de0247c715423 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9ca3bab29b4c4dce908de0247c715423, State: Running, Role: LEADER
I20260807 03:37:44.513983 16364 consensus_queue.cc:260] T 485e8d9a337d4e4f9207f737e3a8812b P 9ca3bab29b4c4dce908de0247c715423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:44.515014 16364 raft_consensus.cc:3037] T 485e8d9a337d4e4f9207f737e3a8812b P 9ca3bab29b4c4dce908de0247c715423 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:37:44.517684 16604 raft_consensus.cc:2454] T 485e8d9a337d4e4f9207f737e3a8812b P ec8c86cff51f48bf9ecc5177e6266b7b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05aa1d36d28a48138040f6c78c4b34a5 in term 2.
I20260807 03:37:44.518882 16433 leader_election.cc:302] T 485e8d9a337d4e4f9207f737e3a8812b P 05aa1d36d28a48138040f6c78c4b34a5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05aa1d36d28a48138040f6c78c4b34a5, ec8c86cff51f48bf9ecc5177e6266b7b; no voters:
I20260807 03:37:44.519583 16908 raft_consensus.cc:2781] T 485e8d9a337d4e4f9207f737e3a8812b P 05aa1d36d28a48138040f6c78c4b34a5 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:37:44.519817 16364 raft_consensus.cc:2454] T 485e8d9a337d4e4f9207f737e3a8812b P 9ca3bab29b4c4dce908de0247c715423 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05aa1d36d28a48138040f6c78c4b34a5 in term 2.
I20260807 03:37:44.520144 16908 raft_consensus.cc:686] T 485e8d9a337d4e4f9207f737e3a8812b P 05aa1d36d28a48138040f6c78c4b34a5 [term 2 LEADER]: Becoming Leader. State: Replica: 05aa1d36d28a48138040f6c78c4b34a5, State: Running, Role: LEADER
I20260807 03:37:44.521026 16908 consensus_queue.cc:237] T 485e8d9a337d4e4f9207f737e3a8812b P 05aa1d36d28a48138040f6c78c4b34a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:44.527891 16207 catalog_manager.cc:5495] T 485e8d9a337d4e4f9207f737e3a8812b P 05aa1d36d28a48138040f6c78c4b34a5 reported cstate change: term changed from 1 to 2, leader changed from 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129) to 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130). New cstate: current_term: 2 leader_uuid: "05aa1d36d28a48138040f6c78c4b34a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } health_report { overall_health: UNKNOWN } } }
I20260807 03:37:44.532357 16861 raft_consensus.cc:979] T f0191ce545e0490986331a9c562b056d P 9ca3bab29b4c4dce908de0247c715423: : Instructing follower 05aa1d36d28a48138040f6c78c4b34a5 to start an election
I20260807 03:37:44.532783 16924 raft_consensus.cc:1067] T f0191ce545e0490986331a9c562b056d P 9ca3bab29b4c4dce908de0247c715423 [term 2 LEADER]: Signalling peer 05aa1d36d28a48138040f6c78c4b34a5 to start an election
I20260807 03:37:44.534127 16479 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f0191ce545e0490986331a9c562b056d"
dest_uuid: "05aa1d36d28a48138040f6c78c4b34a5"
from {username='slave'} at 127.0.0.1:53828
I20260807 03:37:44.534569 16479 raft_consensus.cc:482] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:37:44.534821 16479 raft_consensus.cc:3037] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:37:44.538396 16861 raft_consensus.cc:979] T fe184e38bf0d4e51bf42238d161ac738 P 9ca3bab29b4c4dce908de0247c715423: : Instructing follower 05aa1d36d28a48138040f6c78c4b34a5 to start an election
I20260807 03:37:44.538800 16924 raft_consensus.cc:1067] T fe184e38bf0d4e51bf42238d161ac738 P 9ca3bab29b4c4dce908de0247c715423 [term 2 LEADER]: Signalling peer 05aa1d36d28a48138040f6c78c4b34a5 to start an election
I20260807 03:37:44.540086 16484 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fe184e38bf0d4e51bf42238d161ac738"
dest_uuid: "05aa1d36d28a48138040f6c78c4b34a5"
from {username='slave'} at 127.0.0.1:53828
I20260807 03:37:44.540285 16479 raft_consensus.cc:504] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:44.540637 16484 raft_consensus.cc:482] T fe184e38bf0d4e51bf42238d161ac738 P 05aa1d36d28a48138040f6c78c4b34a5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:37:44.540987 16484 raft_consensus.cc:3037] T fe184e38bf0d4e51bf42238d161ac738 P 05aa1d36d28a48138040f6c78c4b34a5 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:37:44.542342 16479 leader_election.cc:288] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5 [CANDIDATE]: Term 3 election: Requested vote from peers 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129:36375), ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131:37775)
I20260807 03:37:44.542650 16364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0191ce545e0490986331a9c562b056d" candidate_uuid: "05aa1d36d28a48138040f6c78c4b34a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9ca3bab29b4c4dce908de0247c715423"
I20260807 03:37:44.543205 16364 raft_consensus.cc:3032] T f0191ce545e0490986331a9c562b056d P 9ca3bab29b4c4dce908de0247c715423 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 03:37:44.543447 16364 raft_consensus.cc:729] T f0191ce545e0490986331a9c562b056d P 9ca3bab29b4c4dce908de0247c715423 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9ca3bab29b4c4dce908de0247c715423, State: Running, Role: LEADER
I20260807 03:37:44.544075 16364 consensus_queue.cc:260] T f0191ce545e0490986331a9c562b056d P 9ca3bab29b4c4dce908de0247c715423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:44.544548 16604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0191ce545e0490986331a9c562b056d" candidate_uuid: "05aa1d36d28a48138040f6c78c4b34a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b"
I20260807 03:37:44.545218 16364 raft_consensus.cc:3037] T f0191ce545e0490986331a9c562b056d P 9ca3bab29b4c4dce908de0247c715423 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:37:44.546629 16484 raft_consensus.cc:504] T fe184e38bf0d4e51bf42238d161ac738 P 05aa1d36d28a48138040f6c78c4b34a5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:44.548056 16484 leader_election.cc:288] T fe184e38bf0d4e51bf42238d161ac738 P 05aa1d36d28a48138040f6c78c4b34a5 [CANDIDATE]: Term 3 election: Requested vote from peers 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129:36375), ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131:37775)
I20260807 03:37:44.549139 16365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe184e38bf0d4e51bf42238d161ac738" candidate_uuid: "05aa1d36d28a48138040f6c78c4b34a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9ca3bab29b4c4dce908de0247c715423"
I20260807 03:37:44.549582 16365 raft_consensus.cc:3032] T fe184e38bf0d4e51bf42238d161ac738 P 9ca3bab29b4c4dce908de0247c715423 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 03:37:44.549831 16365 raft_consensus.cc:729] T fe184e38bf0d4e51bf42238d161ac738 P 9ca3bab29b4c4dce908de0247c715423 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9ca3bab29b4c4dce908de0247c715423, State: Running, Role: LEADER
I20260807 03:37:44.550209 16364 raft_consensus.cc:2454] T f0191ce545e0490986331a9c562b056d P 9ca3bab29b4c4dce908de0247c715423 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05aa1d36d28a48138040f6c78c4b34a5 in term 3.
I20260807 03:37:44.545239 16604 raft_consensus.cc:3037] T f0191ce545e0490986331a9c562b056d P ec8c86cff51f48bf9ecc5177e6266b7b [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:37:44.550405 16365 consensus_queue.cc:260] T fe184e38bf0d4e51bf42238d161ac738 P 9ca3bab29b4c4dce908de0247c715423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:44.551245 16432 leader_election.cc:302] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05aa1d36d28a48138040f6c78c4b34a5, 9ca3bab29b4c4dce908de0247c715423; no voters:
I20260807 03:37:44.551553 16365 raft_consensus.cc:3037] T fe184e38bf0d4e51bf42238d161ac738 P 9ca3bab29b4c4dce908de0247c715423 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:37:44.552035 16908 raft_consensus.cc:2781] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:37:44.552525 16908 raft_consensus.cc:686] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5 [term 3 LEADER]: Becoming Leader. State: Replica: 05aa1d36d28a48138040f6c78c4b34a5, State: Running, Role: LEADER
I20260807 03:37:44.553872 16908 consensus_queue.cc:237] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:44.555531 16604 raft_consensus.cc:2454] T f0191ce545e0490986331a9c562b056d P ec8c86cff51f48bf9ecc5177e6266b7b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05aa1d36d28a48138040f6c78c4b34a5 in term 3.
I20260807 03:37:44.556456 16606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe184e38bf0d4e51bf42238d161ac738" candidate_uuid: "05aa1d36d28a48138040f6c78c4b34a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b"
I20260807 03:37:44.557014 16606 raft_consensus.cc:3037] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:37:44.558427 16365 raft_consensus.cc:2454] T fe184e38bf0d4e51bf42238d161ac738 P 9ca3bab29b4c4dce908de0247c715423 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05aa1d36d28a48138040f6c78c4b34a5 in term 3.
I20260807 03:37:44.559777 16432 leader_election.cc:302] T fe184e38bf0d4e51bf42238d161ac738 P 05aa1d36d28a48138040f6c78c4b34a5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05aa1d36d28a48138040f6c78c4b34a5, 9ca3bab29b4c4dce908de0247c715423; no voters:
I20260807 03:37:44.560672 16881 raft_consensus.cc:2781] T fe184e38bf0d4e51bf42238d161ac738 P 05aa1d36d28a48138040f6c78c4b34a5 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:37:44.561144 16881 raft_consensus.cc:686] T fe184e38bf0d4e51bf42238d161ac738 P 05aa1d36d28a48138040f6c78c4b34a5 [term 3 LEADER]: Becoming Leader. State: Replica: 05aa1d36d28a48138040f6c78c4b34a5, State: Running, Role: LEADER
I20260807 03:37:44.561959 16881 consensus_queue.cc:237] T fe184e38bf0d4e51bf42238d161ac738 P 05aa1d36d28a48138040f6c78c4b34a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:44.562516 16606 raft_consensus.cc:2454] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05aa1d36d28a48138040f6c78c4b34a5 in term 3.
I20260807 03:37:44.564344 16207 catalog_manager.cc:5495] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5 reported cstate change: term changed from 2 to 3, leader changed from 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129) to 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130). New cstate: current_term: 3 leader_uuid: "05aa1d36d28a48138040f6c78c4b34a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } health_report { overall_health: UNKNOWN } } }
I20260807 03:37:44.574067 16207 catalog_manager.cc:5495] T fe184e38bf0d4e51bf42238d161ac738 P 05aa1d36d28a48138040f6c78c4b34a5 reported cstate change: term changed from 2 to 3, leader changed from 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129) to 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130). New cstate: current_term: 3 leader_uuid: "05aa1d36d28a48138040f6c78c4b34a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } health_report { overall_health: UNKNOWN } } }
I20260807 03:37:44.834254 16365 raft_consensus.cc:1261] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [term 5 FOLLOWER]: Refusing update from remote peer 05aa1d36d28a48138040f6c78c4b34a5: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 03:37:44.835350 16908 consensus_queue.cc:1035] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:37:44.838528 16606 raft_consensus.cc:1261] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b [term 5 FOLLOWER]: Refusing update from remote peer 05aa1d36d28a48138040f6c78c4b34a5: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 03:37:44.839810 16908 consensus_queue.cc:1035] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 03:37:44.910578 16606 raft_consensus.cc:1261] T 485e8d9a337d4e4f9207f737e3a8812b P ec8c86cff51f48bf9ecc5177e6266b7b [term 2 FOLLOWER]: Refusing update from remote peer 05aa1d36d28a48138040f6c78c4b34a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:37:44.911703 16881 consensus_queue.cc:1035] T 485e8d9a337d4e4f9207f737e3a8812b P 05aa1d36d28a48138040f6c78c4b34a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:37:44.919358 16365 raft_consensus.cc:1261] T 485e8d9a337d4e4f9207f737e3a8812b P 9ca3bab29b4c4dce908de0247c715423 [term 2 FOLLOWER]: Refusing update from remote peer 05aa1d36d28a48138040f6c78c4b34a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:37:44.921020 16881 consensus_queue.cc:1035] T 485e8d9a337d4e4f9207f737e3a8812b P 05aa1d36d28a48138040f6c78c4b34a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:37:44.922392 16604 raft_consensus.cc:1261] T 10452db9b82745b68c6c353b25eff5ef P ec8c86cff51f48bf9ecc5177e6266b7b [term 4 FOLLOWER]: Refusing update from remote peer 05aa1d36d28a48138040f6c78c4b34a5: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260807 03:37:44.926441 16939 consensus_queue.cc:1035] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 03:37:44.940601 16364 raft_consensus.cc:1261] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [term 4 FOLLOWER]: Refusing update from remote peer 05aa1d36d28a48138040f6c78c4b34a5: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260807 03:37:44.941663 16881 consensus_queue.cc:1035] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:37:44.975782 16364 raft_consensus.cc:1261] T f0191ce545e0490986331a9c562b056d P 9ca3bab29b4c4dce908de0247c715423 [term 3 FOLLOWER]: Refusing update from remote peer 05aa1d36d28a48138040f6c78c4b34a5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 03:37:44.976799 16881 consensus_queue.cc:1035] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:37:44.986224 16604 raft_consensus.cc:1261] T f0191ce545e0490986331a9c562b056d P ec8c86cff51f48bf9ecc5177e6266b7b [term 3 FOLLOWER]: Refusing update from remote peer 05aa1d36d28a48138040f6c78c4b34a5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 03:37:44.987453 16909 consensus_queue.cc:1035] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:37:45.039191 16364 raft_consensus.cc:1261] T fe184e38bf0d4e51bf42238d161ac738 P 9ca3bab29b4c4dce908de0247c715423 [term 3 FOLLOWER]: Refusing update from remote peer 05aa1d36d28a48138040f6c78c4b34a5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 03:37:45.040525 16909 consensus_queue.cc:1035] T fe184e38bf0d4e51bf42238d161ac738 P 05aa1d36d28a48138040f6c78c4b34a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:37:45.048219 16604 raft_consensus.cc:1261] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b [term 3 FOLLOWER]: Refusing update from remote peer 05aa1d36d28a48138040f6c78c4b34a5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 03:37:45.049602 16939 consensus_queue.cc:1035] T fe184e38bf0d4e51bf42238d161ac738 P 05aa1d36d28a48138040f6c78c4b34a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:37:47.309852 16154 auto_leader_rebalancer-test.cc:303] The leader distribution is
I20260807 03:37:47.310513 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:37:47.317976 16484 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "10452db9b82745b68c6c353b25eff5ef"
dest_uuid: "05aa1d36d28a48138040f6c78c4b34a5"
mode: GRACEFUL
new_leader_uuid: "9ca3bab29b4c4dce908de0247c715423"
from {username='slave'} at 127.0.0.1:41698
I20260807 03:37:47.318921 16484 raft_consensus.cc:595] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [term 4 LEADER]: Received request to transfer leadership to TS 9ca3bab29b4c4dce908de0247c715423
I20260807 03:37:47.321307 16484 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5e409bda14f142959c8127f3bd2643fc"
dest_uuid: "05aa1d36d28a48138040f6c78c4b34a5"
mode: GRACEFUL
new_leader_uuid: "9ca3bab29b4c4dce908de0247c715423"
from {username='slave'} at 127.0.0.1:41698
I20260807 03:37:47.321763 16484 raft_consensus.cc:595] T 5e409bda14f142959c8127f3bd2643fc P 05aa1d36d28a48138040f6c78c4b34a5 [term 2 LEADER]: Received request to transfer leadership to TS 9ca3bab29b4c4dce908de0247c715423
I20260807 03:37:47.323558 16484 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d07d00a26d54438392d5db566a17627b"
dest_uuid: "05aa1d36d28a48138040f6c78c4b34a5"
mode: GRACEFUL
new_leader_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b"
from {username='slave'} at 127.0.0.1:41698
I20260807 03:37:47.323997 16484 raft_consensus.cc:595] T d07d00a26d54438392d5db566a17627b P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 LEADER]: Received request to transfer leadership to TS ec8c86cff51f48bf9ecc5177e6266b7b
I20260807 03:37:47.325881 16484 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e9e0fd324ec84973b604e22e851d7b49"
dest_uuid: "05aa1d36d28a48138040f6c78c4b34a5"
mode: GRACEFUL
new_leader_uuid: "9ca3bab29b4c4dce908de0247c715423"
from {username='slave'} at 127.0.0.1:41698
I20260807 03:37:47.326321 16484 raft_consensus.cc:595] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 5 LEADER]: Received request to transfer leadership to TS 9ca3bab29b4c4dce908de0247c715423
I20260807 03:37:47.328081 16484 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fe184e38bf0d4e51bf42238d161ac738"
dest_uuid: "05aa1d36d28a48138040f6c78c4b34a5"
mode: GRACEFUL
new_leader_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b"
from {username='slave'} at 127.0.0.1:41698
I20260807 03:37:47.328523 16484 raft_consensus.cc:595] T fe184e38bf0d4e51bf42238d161ac738 P 05aa1d36d28a48138040f6c78c4b34a5 [term 3 LEADER]: Received request to transfer leadership to TS ec8c86cff51f48bf9ecc5177e6266b7b
I20260807 03:37:47.329347 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260807 03:37:47.329757 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:37:47.459067 16939 raft_consensus.cc:979] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5: : Instructing follower 9ca3bab29b4c4dce908de0247c715423 to start an election
I20260807 03:37:47.459630 16961 raft_consensus.cc:1067] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 5 LEADER]: Signalling peer 9ca3bab29b4c4dce908de0247c715423 to start an election
I20260807 03:37:47.461021 16365 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e9e0fd324ec84973b604e22e851d7b49"
dest_uuid: "9ca3bab29b4c4dce908de0247c715423"
from {username='slave'} at 127.0.0.1:44368
I20260807 03:37:47.461467 16365 raft_consensus.cc:482] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:37:47.461827 16365 raft_consensus.cc:3037] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:37:47.467270 16365 raft_consensus.cc:504] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:47.469544 16365 leader_election.cc:288] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [CANDIDATE]: Term 6 election: Requested vote from peers 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130:39359), ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131:37775)
I20260807 03:37:47.470378 16484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e0fd324ec84973b604e22e851d7b49" candidate_uuid: "9ca3bab29b4c4dce908de0247c715423" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "05aa1d36d28a48138040f6c78c4b34a5"
I20260807 03:37:47.470575 16604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e0fd324ec84973b604e22e851d7b49" candidate_uuid: "9ca3bab29b4c4dce908de0247c715423" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b"
I20260807 03:37:47.471256 16484 raft_consensus.cc:3032] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 03:37:47.471321 16604 raft_consensus.cc:3037] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:37:47.471494 16484 raft_consensus.cc:729] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 05aa1d36d28a48138040f6c78c4b34a5, State: Running, Role: LEADER
I20260807 03:37:47.472306 16484 consensus_queue.cc:260] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:47.473214 16484 raft_consensus.cc:3037] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:37:47.475520 16604 raft_consensus.cc:2454] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ca3bab29b4c4dce908de0247c715423 in term 6.
I20260807 03:37:47.476578 16314 leader_election.cc:302] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ca3bab29b4c4dce908de0247c715423, ec8c86cff51f48bf9ecc5177e6266b7b; no voters:
I20260807 03:37:47.477537 16484 raft_consensus.cc:2454] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ca3bab29b4c4dce908de0247c715423 in term 6.
I20260807 03:37:47.478060 16962 raft_consensus.cc:2781] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 03:37:47.479051 16962 raft_consensus.cc:686] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [term 6 LEADER]: Becoming Leader. State: Replica: 9ca3bab29b4c4dce908de0247c715423, State: Running, Role: LEADER
I20260807 03:37:47.479871 16962 consensus_queue.cc:237] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:47.486007 16206 catalog_manager.cc:5495] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 reported cstate change: term changed from 5 to 6, leader changed from 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130) to 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129). New cstate: current_term: 6 leader_uuid: "9ca3bab29b4c4dce908de0247c715423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } health_report { overall_health: UNKNOWN } } }
I20260807 03:37:47.502308 16939 raft_consensus.cc:979] T 5e409bda14f142959c8127f3bd2643fc P 05aa1d36d28a48138040f6c78c4b34a5: : Instructing follower 9ca3bab29b4c4dce908de0247c715423 to start an election
I20260807 03:37:47.502707 16961 raft_consensus.cc:1067] T 5e409bda14f142959c8127f3bd2643fc P 05aa1d36d28a48138040f6c78c4b34a5 [term 2 LEADER]: Signalling peer 9ca3bab29b4c4dce908de0247c715423 to start an election
I20260807 03:37:47.503970 16365 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5e409bda14f142959c8127f3bd2643fc"
dest_uuid: "9ca3bab29b4c4dce908de0247c715423"
from {username='slave'} at 127.0.0.1:44368
I20260807 03:37:47.504468 16365 raft_consensus.cc:482] T 5e409bda14f142959c8127f3bd2643fc P 9ca3bab29b4c4dce908de0247c715423 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:37:47.504732 16365 raft_consensus.cc:3037] T 5e409bda14f142959c8127f3bd2643fc P 9ca3bab29b4c4dce908de0247c715423 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:37:47.509039 16365 raft_consensus.cc:504] T 5e409bda14f142959c8127f3bd2643fc P 9ca3bab29b4c4dce908de0247c715423 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:47.510486 16365 leader_election.cc:288] T 5e409bda14f142959c8127f3bd2643fc P 9ca3bab29b4c4dce908de0247c715423 [CANDIDATE]: Term 3 election: Requested vote from peers 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130:39359), ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131:37775)
I20260807 03:37:47.511111 16484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e409bda14f142959c8127f3bd2643fc" candidate_uuid: "9ca3bab29b4c4dce908de0247c715423" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "05aa1d36d28a48138040f6c78c4b34a5"
I20260807 03:37:47.511324 16604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e409bda14f142959c8127f3bd2643fc" candidate_uuid: "9ca3bab29b4c4dce908de0247c715423" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b"
I20260807 03:37:47.511683 16484 raft_consensus.cc:3032] T 5e409bda14f142959c8127f3bd2643fc P 05aa1d36d28a48138040f6c78c4b34a5 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 03:37:47.511782 16604 raft_consensus.cc:3037] T 5e409bda14f142959c8127f3bd2643fc P ec8c86cff51f48bf9ecc5177e6266b7b [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:37:47.511993 16484 raft_consensus.cc:729] T 5e409bda14f142959c8127f3bd2643fc P 05aa1d36d28a48138040f6c78c4b34a5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 05aa1d36d28a48138040f6c78c4b34a5, State: Running, Role: LEADER
I20260807 03:37:47.512764 16484 consensus_queue.cc:260] T 5e409bda14f142959c8127f3bd2643fc P 05aa1d36d28a48138040f6c78c4b34a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:47.513865 16484 raft_consensus.cc:3037] T 5e409bda14f142959c8127f3bd2643fc P 05aa1d36d28a48138040f6c78c4b34a5 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:37:47.517441 16604 raft_consensus.cc:2454] T 5e409bda14f142959c8127f3bd2643fc P ec8c86cff51f48bf9ecc5177e6266b7b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ca3bab29b4c4dce908de0247c715423 in term 3.
I20260807 03:37:47.518399 16314 leader_election.cc:302] T 5e409bda14f142959c8127f3bd2643fc P 9ca3bab29b4c4dce908de0247c715423 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ca3bab29b4c4dce908de0247c715423, ec8c86cff51f48bf9ecc5177e6266b7b; no voters:
I20260807 03:37:47.519124 16962 raft_consensus.cc:2781] T 5e409bda14f142959c8127f3bd2643fc P 9ca3bab29b4c4dce908de0247c715423 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:37:47.519546 16962 raft_consensus.cc:686] T 5e409bda14f142959c8127f3bd2643fc P 9ca3bab29b4c4dce908de0247c715423 [term 3 LEADER]: Becoming Leader. State: Replica: 9ca3bab29b4c4dce908de0247c715423, State: Running, Role: LEADER
I20260807 03:37:47.519466 16484 raft_consensus.cc:2454] T 5e409bda14f142959c8127f3bd2643fc P 05aa1d36d28a48138040f6c78c4b34a5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ca3bab29b4c4dce908de0247c715423 in term 3.
I20260807 03:37:47.520550 16962 consensus_queue.cc:237] T 5e409bda14f142959c8127f3bd2643fc P 9ca3bab29b4c4dce908de0247c715423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:47.527132 16206 catalog_manager.cc:5495] T 5e409bda14f142959c8127f3bd2643fc P 9ca3bab29b4c4dce908de0247c715423 reported cstate change: term changed from 2 to 3, leader changed from 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130) to 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129). New cstate: current_term: 3 leader_uuid: "9ca3bab29b4c4dce908de0247c715423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } health_report { overall_health: UNKNOWN } } }
I20260807 03:37:47.577747 16939 raft_consensus.cc:979] T fe184e38bf0d4e51bf42238d161ac738 P 05aa1d36d28a48138040f6c78c4b34a5: : Instructing follower ec8c86cff51f48bf9ecc5177e6266b7b to start an election
I20260807 03:37:47.578147 16961 raft_consensus.cc:1067] T fe184e38bf0d4e51bf42238d161ac738 P 05aa1d36d28a48138040f6c78c4b34a5 [term 3 LEADER]: Signalling peer ec8c86cff51f48bf9ecc5177e6266b7b to start an election
I20260807 03:37:47.579476 16604 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fe184e38bf0d4e51bf42238d161ac738"
dest_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b"
from {username='slave'} at 127.0.0.1:45596
I20260807 03:37:47.580051 16604 raft_consensus.cc:482] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:37:47.580341 16604 raft_consensus.cc:3037] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:37:47.584307 16604 raft_consensus.cc:504] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:47.585954 16604 leader_election.cc:288] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b [CANDIDATE]: Term 4 election: Requested vote from peers 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129:36375), 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130:39359)
I20260807 03:37:47.586889 16365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe184e38bf0d4e51bf42238d161ac738" candidate_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "9ca3bab29b4c4dce908de0247c715423"
I20260807 03:37:47.586953 16484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe184e38bf0d4e51bf42238d161ac738" candidate_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "05aa1d36d28a48138040f6c78c4b34a5"
I20260807 03:37:47.587462 16365 raft_consensus.cc:3037] T fe184e38bf0d4e51bf42238d161ac738 P 9ca3bab29b4c4dce908de0247c715423 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:37:47.587517 16484 raft_consensus.cc:3032] T fe184e38bf0d4e51bf42238d161ac738 P 05aa1d36d28a48138040f6c78c4b34a5 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:37:47.587963 16484 raft_consensus.cc:729] T fe184e38bf0d4e51bf42238d161ac738 P 05aa1d36d28a48138040f6c78c4b34a5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 05aa1d36d28a48138040f6c78c4b34a5, State: Running, Role: LEADER
I20260807 03:37:47.588644 16484 consensus_queue.cc:260] T fe184e38bf0d4e51bf42238d161ac738 P 05aa1d36d28a48138040f6c78c4b34a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:47.589751 16484 raft_consensus.cc:3037] T fe184e38bf0d4e51bf42238d161ac738 P 05aa1d36d28a48138040f6c78c4b34a5 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:37:47.594538 16365 raft_consensus.cc:2454] T fe184e38bf0d4e51bf42238d161ac738 P 9ca3bab29b4c4dce908de0247c715423 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec8c86cff51f48bf9ecc5177e6266b7b in term 4.
I20260807 03:37:47.595019 16484 raft_consensus.cc:2454] T fe184e38bf0d4e51bf42238d161ac738 P 05aa1d36d28a48138040f6c78c4b34a5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec8c86cff51f48bf9ecc5177e6266b7b in term 4.
I20260807 03:37:47.595777 16556 leader_election.cc:302] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ca3bab29b4c4dce908de0247c715423, ec8c86cff51f48bf9ecc5177e6266b7b; no voters:
I20260807 03:37:47.596737 16941 raft_consensus.cc:2781] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b [term 4 FOLLOWER]: Leader election won for term 4
I20260807 03:37:47.597769 16941 raft_consensus.cc:686] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b [term 4 LEADER]: Becoming Leader. State: Replica: ec8c86cff51f48bf9ecc5177e6266b7b, State: Running, Role: LEADER
I20260807 03:37:47.598424 16941 consensus_queue.cc:237] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:47.605183 16206 catalog_manager.cc:5495] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b reported cstate change: term changed from 3 to 4, leader changed from 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130) to ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131). New cstate: current_term: 4 leader_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } health_report { overall_health: HEALTHY } } }
I20260807 03:37:47.687863 16939 raft_consensus.cc:979] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5: : Instructing follower 9ca3bab29b4c4dce908de0247c715423 to start an election
I20260807 03:37:47.688377 16961 raft_consensus.cc:1067] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [term 4 LEADER]: Signalling peer 9ca3bab29b4c4dce908de0247c715423 to start an election
I20260807 03:37:47.689640 16365 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "10452db9b82745b68c6c353b25eff5ef"
dest_uuid: "9ca3bab29b4c4dce908de0247c715423"
from {username='slave'} at 127.0.0.1:44368
I20260807 03:37:47.690097 16365 raft_consensus.cc:482] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:37:47.690364 16365 raft_consensus.cc:3037] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:37:47.694885 16365 raft_consensus.cc:504] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:47.696424 16365 leader_election.cc:288] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [CANDIDATE]: Term 5 election: Requested vote from peers 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130:39359), ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131:37775)
I20260807 03:37:47.697140 16484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10452db9b82745b68c6c353b25eff5ef" candidate_uuid: "9ca3bab29b4c4dce908de0247c715423" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "05aa1d36d28a48138040f6c78c4b34a5"
I20260807 03:37:47.697436 16604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10452db9b82745b68c6c353b25eff5ef" candidate_uuid: "9ca3bab29b4c4dce908de0247c715423" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b"
I20260807 03:37:47.697660 16484 raft_consensus.cc:3032] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 03:37:47.697944 16604 raft_consensus.cc:3037] T 10452db9b82745b68c6c353b25eff5ef P ec8c86cff51f48bf9ecc5177e6266b7b [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:37:47.697970 16484 raft_consensus.cc:729] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 05aa1d36d28a48138040f6c78c4b34a5, State: Running, Role: LEADER
I20260807 03:37:47.698671 16484 consensus_queue.cc:260] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:47.699545 16484 raft_consensus.cc:3037] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:37:47.701953 16604 raft_consensus.cc:2454] T 10452db9b82745b68c6c353b25eff5ef P ec8c86cff51f48bf9ecc5177e6266b7b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ca3bab29b4c4dce908de0247c715423 in term 5.
I20260807 03:37:47.702677 16314 leader_election.cc:302] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ca3bab29b4c4dce908de0247c715423, ec8c86cff51f48bf9ecc5177e6266b7b; no voters:
I20260807 03:37:47.703279 16962 raft_consensus.cc:2781] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 03:37:47.703642 16484 raft_consensus.cc:2454] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ca3bab29b4c4dce908de0247c715423 in term 5.
I20260807 03:37:47.703707 16962 raft_consensus.cc:686] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [term 5 LEADER]: Becoming Leader. State: Replica: 9ca3bab29b4c4dce908de0247c715423, State: Running, Role: LEADER
I20260807 03:37:47.704645 16962 consensus_queue.cc:237] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:47.710453 16206 catalog_manager.cc:5495] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 reported cstate change: term changed from 4 to 5, leader changed from 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130) to 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129). New cstate: current_term: 5 leader_uuid: "9ca3bab29b4c4dce908de0247c715423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } health_report { overall_health: UNKNOWN } } }
I20260807 03:37:47.929136 16484 raft_consensus.cc:1261] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 6 FOLLOWER]: Refusing update from remote peer 9ca3bab29b4c4dce908de0247c715423: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260807 03:37:47.930358 16962 consensus_queue.cc:1035] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 03:37:47.941119 16604 raft_consensus.cc:1261] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b [term 6 FOLLOWER]: Refusing update from remote peer 9ca3bab29b4c4dce908de0247c715423: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260807 03:37:47.942660 16962 consensus_queue.cc:1035] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 03:37:47.958277 16984 raft_consensus.cc:979] T d07d00a26d54438392d5db566a17627b P 05aa1d36d28a48138040f6c78c4b34a5: : Instructing follower ec8c86cff51f48bf9ecc5177e6266b7b to start an election
I20260807 03:37:47.959780 16939 raft_consensus.cc:1067] T d07d00a26d54438392d5db566a17627b P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 LEADER]: Signalling peer ec8c86cff51f48bf9ecc5177e6266b7b to start an election
I20260807 03:37:47.961131 16604 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d07d00a26d54438392d5db566a17627b"
dest_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b"
from {username='slave'} at 127.0.0.1:45596
I20260807 03:37:47.961630 16604 raft_consensus.cc:482] T d07d00a26d54438392d5db566a17627b P ec8c86cff51f48bf9ecc5177e6266b7b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:37:47.961877 16604 raft_consensus.cc:3037] T d07d00a26d54438392d5db566a17627b P ec8c86cff51f48bf9ecc5177e6266b7b [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:37:47.967008 16604 raft_consensus.cc:504] T d07d00a26d54438392d5db566a17627b P ec8c86cff51f48bf9ecc5177e6266b7b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:47.968412 16604 leader_election.cc:288] T d07d00a26d54438392d5db566a17627b P ec8c86cff51f48bf9ecc5177e6266b7b [CANDIDATE]: Term 2 election: Requested vote from peers 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129:36375), 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130:39359)
I20260807 03:37:47.969094 16365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d07d00a26d54438392d5db566a17627b" candidate_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9ca3bab29b4c4dce908de0247c715423"
I20260807 03:37:47.969435 16484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d07d00a26d54438392d5db566a17627b" candidate_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "05aa1d36d28a48138040f6c78c4b34a5"
I20260807 03:37:47.969657 16365 raft_consensus.cc:3037] T d07d00a26d54438392d5db566a17627b P 9ca3bab29b4c4dce908de0247c715423 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:37:47.970072 16484 raft_consensus.cc:3032] T d07d00a26d54438392d5db566a17627b P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:37:47.970319 16484 raft_consensus.cc:729] T d07d00a26d54438392d5db566a17627b P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 05aa1d36d28a48138040f6c78c4b34a5, State: Running, Role: LEADER
I20260807 03:37:47.970882 16484 consensus_queue.cc:260] T d07d00a26d54438392d5db566a17627b P 05aa1d36d28a48138040f6c78c4b34a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:47.971692 16484 raft_consensus.cc:3037] T d07d00a26d54438392d5db566a17627b P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:37:47.973786 16365 raft_consensus.cc:2454] T d07d00a26d54438392d5db566a17627b P 9ca3bab29b4c4dce908de0247c715423 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec8c86cff51f48bf9ecc5177e6266b7b in term 2.
I20260807 03:37:47.974630 16556 leader_election.cc:302] T d07d00a26d54438392d5db566a17627b P ec8c86cff51f48bf9ecc5177e6266b7b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ca3bab29b4c4dce908de0247c715423, ec8c86cff51f48bf9ecc5177e6266b7b; no voters:
I20260807 03:37:47.975275 16941 raft_consensus.cc:2781] T d07d00a26d54438392d5db566a17627b P ec8c86cff51f48bf9ecc5177e6266b7b [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:37:47.975793 16941 raft_consensus.cc:686] T d07d00a26d54438392d5db566a17627b P ec8c86cff51f48bf9ecc5177e6266b7b [term 2 LEADER]: Becoming Leader. State: Replica: ec8c86cff51f48bf9ecc5177e6266b7b, State: Running, Role: LEADER
I20260807 03:37:47.975844 16484 raft_consensus.cc:2454] T d07d00a26d54438392d5db566a17627b P 05aa1d36d28a48138040f6c78c4b34a5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec8c86cff51f48bf9ecc5177e6266b7b in term 2.
I20260807 03:37:47.976658 16941 consensus_queue.cc:237] T d07d00a26d54438392d5db566a17627b P ec8c86cff51f48bf9ecc5177e6266b7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } }
I20260807 03:37:47.983911 16208 catalog_manager.cc:5495] T d07d00a26d54438392d5db566a17627b P ec8c86cff51f48bf9ecc5177e6266b7b reported cstate change: term changed from 1 to 2, leader changed from 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130) to ec8c86cff51f48bf9ecc5177e6266b7b (127.15.198.131). New cstate: current_term: 2 leader_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 } health_report { overall_health: HEALTHY } } }
I20260807 03:37:48.078909 16484 raft_consensus.cc:1261] T 5e409bda14f142959c8127f3bd2643fc P 05aa1d36d28a48138040f6c78c4b34a5 [term 3 FOLLOWER]: Refusing update from remote peer 9ca3bab29b4c4dce908de0247c715423: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 03:37:48.080261 16962 consensus_queue.cc:1035] T 5e409bda14f142959c8127f3bd2643fc P 9ca3bab29b4c4dce908de0247c715423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:37:48.088580 16604 raft_consensus.cc:1261] T 5e409bda14f142959c8127f3bd2643fc P ec8c86cff51f48bf9ecc5177e6266b7b [term 3 FOLLOWER]: Refusing update from remote peer 9ca3bab29b4c4dce908de0247c715423: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 03:37:48.090412 16963 consensus_queue.cc:1035] T 5e409bda14f142959c8127f3bd2643fc P 9ca3bab29b4c4dce908de0247c715423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:37:48.112602 16484 raft_consensus.cc:1261] T fe184e38bf0d4e51bf42238d161ac738 P 05aa1d36d28a48138040f6c78c4b34a5 [term 4 FOLLOWER]: Refusing update from remote peer ec8c86cff51f48bf9ecc5177e6266b7b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260807 03:37:48.113531 16941 consensus_queue.cc:1035] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:37:48.121490 16365 raft_consensus.cc:1261] T fe184e38bf0d4e51bf42238d161ac738 P 9ca3bab29b4c4dce908de0247c715423 [term 4 FOLLOWER]: Refusing update from remote peer ec8c86cff51f48bf9ecc5177e6266b7b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260807 03:37:48.122920 16941 consensus_queue.cc:1035] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:37:48.157014 16484 raft_consensus.cc:1261] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [term 5 FOLLOWER]: Refusing update from remote peer 9ca3bab29b4c4dce908de0247c715423: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260807 03:37:48.158010 16963 consensus_queue.cc:1035] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 03:37:48.165606 16604 raft_consensus.cc:1261] T 10452db9b82745b68c6c353b25eff5ef P ec8c86cff51f48bf9ecc5177e6266b7b [term 5 FOLLOWER]: Refusing update from remote peer 9ca3bab29b4c4dce908de0247c715423: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260807 03:37:48.167007 16992 consensus_queue.cc:1035] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8c86cff51f48bf9ecc5177e6266b7b" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 37775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 03:37:48.330583 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:37:48.335664 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 03:37:48.336265 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:37:48.430053 16484 raft_consensus.cc:1261] T d07d00a26d54438392d5db566a17627b P 05aa1d36d28a48138040f6c78c4b34a5 [term 2 FOLLOWER]: Refusing update from remote peer ec8c86cff51f48bf9ecc5177e6266b7b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:37:48.431282 16941 consensus_queue.cc:1035] T d07d00a26d54438392d5db566a17627b P ec8c86cff51f48bf9ecc5177e6266b7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "05aa1d36d28a48138040f6c78c4b34a5" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:37:48.440526 16365 raft_consensus.cc:1261] T d07d00a26d54438392d5db566a17627b P 9ca3bab29b4c4dce908de0247c715423 [term 2 FOLLOWER]: Refusing update from remote peer ec8c86cff51f48bf9ecc5177e6266b7b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:37:48.442375 16995 consensus_queue.cc:1035] T d07d00a26d54438392d5db566a17627b P ec8c86cff51f48bf9ecc5177e6266b7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ca3bab29b4c4dce908de0247c715423" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260807 03:37:49.536896 16433 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:37775, user_credentials={real_user=slave}} blocked reactor thread for 44018.1us
I20260807 03:37:49.629719 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:37:49.658725 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 03:37:49.659255 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260807 03:37:50.558842 16404 debug-util.cc:398] Leaking SignalData structure 0x7b08000e9060 after lost signal to thread 16157
W20260807 03:37:50.559640 16404 debug-util.cc:398] Leaking SignalData structure 0x7b08000e9740 after lost signal to thread 16262
W20260807 03:37:50.560496 16404 debug-util.cc:398] Leaking SignalData structure 0x7b08000c5da0 after lost signal to thread 16407
W20260807 03:37:50.561162 16404 debug-util.cc:398] Leaking SignalData structure 0x7b08001e5d00 after lost signal to thread 16525
W20260807 03:37:50.562160 16404 debug-util.cc:398] Leaking SignalData structure 0x7b08000ad2e0 after lost signal to thread 16644
I20260807 03:37:50.660210 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:37:50.666357 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 03:37:50.666817 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:37:51.667774 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:37:51.673529 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 03:37:51.673931 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:37:52.674659 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:37:52.679662 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 03:37:52.680043 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:37:53.680943 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:37:53.685776 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 03:37:53.686093 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:37:54.686797 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:37:54.693724 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 03:37:54.694036 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:37:55.694685 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:37:55.699247 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 03:37:55.699561 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:37:56.700786 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:37:56.706307 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 03:37:56.706653 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:37:57.732684 16154 auto_leader_rebalancer-test.cc:314] The leader distribution is
I20260807 03:37:57.733227 16154 tablet_server.cc:178] TabletServer@127.15.198.129:0 shutting down...
I20260807 03:37:57.769501 16154 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260807 03:37:57.770133 16154 tablet_replica.cc:330] T a4e0d750de56427e8a0af5bc030271c4 P 9ca3bab29b4c4dce908de0247c715423: stopping tablet replica
I20260807 03:37:57.770991 16154 raft_consensus.cc:2229] T a4e0d750de56427e8a0af5bc030271c4 P 9ca3bab29b4c4dce908de0247c715423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:37:57.771652 16154 raft_consensus.cc:2258] T a4e0d750de56427e8a0af5bc030271c4 P 9ca3bab29b4c4dce908de0247c715423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:37:57.774248 16154 tablet_replica.cc:330] T d07d00a26d54438392d5db566a17627b P 9ca3bab29b4c4dce908de0247c715423: stopping tablet replica
I20260807 03:37:57.774777 16154 raft_consensus.cc:2229] T d07d00a26d54438392d5db566a17627b P 9ca3bab29b4c4dce908de0247c715423 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260807 03:37:57.775213 16154 raft_consensus.cc:2258] T d07d00a26d54438392d5db566a17627b P 9ca3bab29b4c4dce908de0247c715423 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260807 03:37:57.776737 16154 tablet_replica.cc:330] T fe184e38bf0d4e51bf42238d161ac738 P 9ca3bab29b4c4dce908de0247c715423: stopping tablet replica
I20260807 03:37:57.777232 16154 raft_consensus.cc:2229] T fe184e38bf0d4e51bf42238d161ac738 P 9ca3bab29b4c4dce908de0247c715423 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260807 03:37:57.777635 16154 raft_consensus.cc:2258] T fe184e38bf0d4e51bf42238d161ac738 P 9ca3bab29b4c4dce908de0247c715423 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260807 03:37:57.779176 16154 tablet_replica.cc:330] T 5e409bda14f142959c8127f3bd2643fc P 9ca3bab29b4c4dce908de0247c715423: stopping tablet replica
I20260807 03:37:57.779636 16154 raft_consensus.cc:2229] T 5e409bda14f142959c8127f3bd2643fc P 9ca3bab29b4c4dce908de0247c715423 [term 3 LEADER]: Raft consensus shutting down.
I20260807 03:37:57.780514 16154 raft_consensus.cc:2258] T 5e409bda14f142959c8127f3bd2643fc P 9ca3bab29b4c4dce908de0247c715423 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 03:37:57.782114 16154 tablet_replica.cc:330] T 242a724d826340b693e5b0767a1ba28c P 9ca3bab29b4c4dce908de0247c715423: stopping tablet replica
I20260807 03:37:57.782589 16154 raft_consensus.cc:2229] T 242a724d826340b693e5b0767a1ba28c P 9ca3bab29b4c4dce908de0247c715423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:37:57.782954 16154 raft_consensus.cc:2258] T 242a724d826340b693e5b0767a1ba28c P 9ca3bab29b4c4dce908de0247c715423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:37:57.784457 16154 tablet_replica.cc:330] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423: stopping tablet replica
I20260807 03:37:57.784919 16154 raft_consensus.cc:2229] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [term 5 LEADER]: Raft consensus shutting down.
I20260807 03:37:57.785629 16154 raft_consensus.cc:2258] T 10452db9b82745b68c6c353b25eff5ef P 9ca3bab29b4c4dce908de0247c715423 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260807 03:37:57.787278 16154 tablet_replica.cc:330] T 485e8d9a337d4e4f9207f737e3a8812b P 9ca3bab29b4c4dce908de0247c715423: stopping tablet replica
I20260807 03:37:57.787735 16154 raft_consensus.cc:2229] T 485e8d9a337d4e4f9207f737e3a8812b P 9ca3bab29b4c4dce908de0247c715423 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260807 03:37:57.788201 16154 raft_consensus.cc:2258] T 485e8d9a337d4e4f9207f737e3a8812b P 9ca3bab29b4c4dce908de0247c715423 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260807 03:37:57.789992 16154 tablet_replica.cc:330] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423: stopping tablet replica
I20260807 03:37:57.790465 16154 raft_consensus.cc:2229] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [term 6 LEADER]: Raft consensus shutting down.
I20260807 03:37:57.791185 16154 raft_consensus.cc:2258] T e9e0fd324ec84973b604e22e851d7b49 P 9ca3bab29b4c4dce908de0247c715423 [term 6 FOLLOWER]: Raft consensus is shut down!
I20260807 03:37:57.793349 16154 tablet_replica.cc:330] T f0191ce545e0490986331a9c562b056d P 9ca3bab29b4c4dce908de0247c715423: stopping tablet replica
I20260807 03:37:57.793789 16154 raft_consensus.cc:2229] T f0191ce545e0490986331a9c562b056d P 9ca3bab29b4c4dce908de0247c715423 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260807 03:37:57.794317 16154 raft_consensus.cc:2258] T f0191ce545e0490986331a9c562b056d P 9ca3bab29b4c4dce908de0247c715423 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260807 03:37:57.900918 16556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.198.129:36375: connect: Connection refused (error 111)
W20260807 03:37:57.905220 16556 consensus_peers.cc:487] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b -> Peer 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129:36375): Couldn't send request to peer 9ca3bab29b4c4dce908de0247c715423. Status: Network error: Client connection negotiation failed: client connection to 127.15.198.129:36375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260807 03:37:57.905751 16556 consensus_peers.cc:487] T d07d00a26d54438392d5db566a17627b P ec8c86cff51f48bf9ecc5177e6266b7b -> Peer 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129:36375): Couldn't send request to peer 9ca3bab29b4c4dce908de0247c715423. Status: Network error: Client connection negotiation failed: client connection to 127.15.198.129:36375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260807 03:37:57.911489 16432 consensus_peers.cc:487] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5 -> Peer 9ca3bab29b4c4dce908de0247c715423 (127.15.198.129:36375): Couldn't send request to peer 9ca3bab29b4c4dce908de0247c715423. Status: Network error: Client connection negotiation failed: client connection to 127.15.198.129:36375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 03:37:57.922570 16154 tablet_server.cc:194] TabletServer@127.15.198.129:0 shutdown complete.
I20260807 03:37:57.943186 16154 tablet_server.cc:178] TabletServer@127.15.198.130:0 shutting down...
I20260807 03:37:57.971983 16154 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260807 03:37:57.972657 16154 tablet_replica.cc:330] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5: stopping tablet replica
I20260807 03:37:57.973556 16154 raft_consensus.cc:2229] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5 [term 3 LEADER]: Raft consensus shutting down.
I20260807 03:37:57.974773 16154 raft_consensus.cc:2258] T f0191ce545e0490986331a9c562b056d P 05aa1d36d28a48138040f6c78c4b34a5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 03:37:57.976524 16154 tablet_replica.cc:330] T 485e8d9a337d4e4f9207f737e3a8812b P 05aa1d36d28a48138040f6c78c4b34a5: stopping tablet replica
I20260807 03:37:57.976995 16154 raft_consensus.cc:2229] T 485e8d9a337d4e4f9207f737e3a8812b P 05aa1d36d28a48138040f6c78c4b34a5 [term 2 LEADER]: Raft consensus shutting down.
I20260807 03:37:57.977667 16154 raft_consensus.cc:2258] T 485e8d9a337d4e4f9207f737e3a8812b P 05aa1d36d28a48138040f6c78c4b34a5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260807 03:37:57.979202 16154 tablet_replica.cc:330] T 5e409bda14f142959c8127f3bd2643fc P 05aa1d36d28a48138040f6c78c4b34a5: stopping tablet replica
I20260807 03:37:57.979713 16154 raft_consensus.cc:2229] T 5e409bda14f142959c8127f3bd2643fc P 05aa1d36d28a48138040f6c78c4b34a5 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260807 03:37:57.980074 16154 raft_consensus.cc:2258] T 5e409bda14f142959c8127f3bd2643fc P 05aa1d36d28a48138040f6c78c4b34a5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 03:37:57.981666 16154 tablet_replica.cc:330] T fe184e38bf0d4e51bf42238d161ac738 P 05aa1d36d28a48138040f6c78c4b34a5: stopping tablet replica
I20260807 03:37:57.982092 16154 raft_consensus.cc:2229] T fe184e38bf0d4e51bf42238d161ac738 P 05aa1d36d28a48138040f6c78c4b34a5 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260807 03:37:57.982510 16154 raft_consensus.cc:2258] T fe184e38bf0d4e51bf42238d161ac738 P 05aa1d36d28a48138040f6c78c4b34a5 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260807 03:37:57.983952 16154 tablet_replica.cc:330] T d07d00a26d54438392d5db566a17627b P 05aa1d36d28a48138040f6c78c4b34a5: stopping tablet replica
I20260807 03:37:57.984434 16154 raft_consensus.cc:2229] T d07d00a26d54438392d5db566a17627b P 05aa1d36d28a48138040f6c78c4b34a5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260807 03:37:57.984771 16154 raft_consensus.cc:2258] T d07d00a26d54438392d5db566a17627b P 05aa1d36d28a48138040f6c78c4b34a5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260807 03:37:57.986243 16154 tablet_replica.cc:330] T a4e0d750de56427e8a0af5bc030271c4 P 05aa1d36d28a48138040f6c78c4b34a5: stopping tablet replica
I20260807 03:37:57.986675 16154 raft_consensus.cc:2229] T a4e0d750de56427e8a0af5bc030271c4 P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 LEADER]: Raft consensus shutting down.
I20260807 03:37:57.987399 16154 raft_consensus.cc:2258] T a4e0d750de56427e8a0af5bc030271c4 P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:37:57.989180 16154 tablet_replica.cc:330] T 242a724d826340b693e5b0767a1ba28c P 05aa1d36d28a48138040f6c78c4b34a5: stopping tablet replica
I20260807 03:37:57.989637 16154 raft_consensus.cc:2229] T 242a724d826340b693e5b0767a1ba28c P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:37:57.989970 16154 raft_consensus.cc:2258] T 242a724d826340b693e5b0767a1ba28c P 05aa1d36d28a48138040f6c78c4b34a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:37:57.991387 16154 tablet_replica.cc:330] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5: stopping tablet replica
I20260807 03:37:57.991809 16154 raft_consensus.cc:2229] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260807 03:37:57.992245 16154 raft_consensus.cc:2258] T 10452db9b82745b68c6c353b25eff5ef P 05aa1d36d28a48138040f6c78c4b34a5 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260807 03:37:57.993620 16154 tablet_replica.cc:330] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5: stopping tablet replica
I20260807 03:37:57.993979 16154 raft_consensus.cc:2229] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 6 FOLLOWER]: Raft consensus shutting down.
I20260807 03:37:57.994361 16154 raft_consensus.cc:2258] T e9e0fd324ec84973b604e22e851d7b49 P 05aa1d36d28a48138040f6c78c4b34a5 [term 6 FOLLOWER]: Raft consensus is shut down!
I20260807 03:37:58.023126 16154 tablet_server.cc:194] TabletServer@127.15.198.130:0 shutdown complete.
I20260807 03:37:58.040400 16154 tablet_server.cc:178] TabletServer@127.15.198.131:0 shutting down...
I20260807 03:37:58.069240 16154 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260807 03:37:58.069980 16154 tablet_replica.cc:330] T 242a724d826340b693e5b0767a1ba28c P ec8c86cff51f48bf9ecc5177e6266b7b: stopping tablet replica
I20260807 03:37:58.070822 16154 raft_consensus.cc:2229] T 242a724d826340b693e5b0767a1ba28c P ec8c86cff51f48bf9ecc5177e6266b7b [term 1 LEADER]: Raft consensus shutting down.
I20260807 03:37:58.071980 16154 raft_consensus.cc:2258] T 242a724d826340b693e5b0767a1ba28c P ec8c86cff51f48bf9ecc5177e6266b7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:37:58.073697 16154 tablet_replica.cc:330] T 10452db9b82745b68c6c353b25eff5ef P ec8c86cff51f48bf9ecc5177e6266b7b: stopping tablet replica
I20260807 03:37:58.074172 16154 raft_consensus.cc:2229] T 10452db9b82745b68c6c353b25eff5ef P ec8c86cff51f48bf9ecc5177e6266b7b [term 5 FOLLOWER]: Raft consensus shutting down.
I20260807 03:37:58.074560 16154 raft_consensus.cc:2258] T 10452db9b82745b68c6c353b25eff5ef P ec8c86cff51f48bf9ecc5177e6266b7b [term 5 FOLLOWER]: Raft consensus is shut down!
I20260807 03:37:58.075995 16154 tablet_replica.cc:330] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b: stopping tablet replica
I20260807 03:37:58.076468 16154 raft_consensus.cc:2229] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b [term 6 FOLLOWER]: Raft consensus shutting down.
I20260807 03:37:58.076864 16154 raft_consensus.cc:2258] T e9e0fd324ec84973b604e22e851d7b49 P ec8c86cff51f48bf9ecc5177e6266b7b [term 6 FOLLOWER]: Raft consensus is shut down!
I20260807 03:37:58.078599 16154 tablet_replica.cc:330] T f0191ce545e0490986331a9c562b056d P ec8c86cff51f48bf9ecc5177e6266b7b: stopping tablet replica
I20260807 03:37:58.079104 16154 raft_consensus.cc:2229] T f0191ce545e0490986331a9c562b056d P ec8c86cff51f48bf9ecc5177e6266b7b [term 3 FOLLOWER]: Raft consensus shutting down.
I20260807 03:37:58.079528 16154 raft_consensus.cc:2258] T f0191ce545e0490986331a9c562b056d P ec8c86cff51f48bf9ecc5177e6266b7b [term 3 FOLLOWER]: Raft consensus is shut down!
W20260807 03:37:58.080581 16557 consensus_peers.cc:487] T d07d00a26d54438392d5db566a17627b P ec8c86cff51f48bf9ecc5177e6266b7b -> Peer 05aa1d36d28a48138040f6c78c4b34a5 (127.15.198.130:39359): Couldn't send request to peer 05aa1d36d28a48138040f6c78c4b34a5. Status: Network error: Client connection negotiation failed: client connection to 127.15.198.130:39359: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 03:37:58.081264 16154 tablet_replica.cc:330] T d07d00a26d54438392d5db566a17627b P ec8c86cff51f48bf9ecc5177e6266b7b: stopping tablet replica
I20260807 03:37:58.081740 16154 raft_consensus.cc:2229] T d07d00a26d54438392d5db566a17627b P ec8c86cff51f48bf9ecc5177e6266b7b [term 2 LEADER]: Raft consensus shutting down.
I20260807 03:37:58.082459 16154 raft_consensus.cc:2258] T d07d00a26d54438392d5db566a17627b P ec8c86cff51f48bf9ecc5177e6266b7b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260807 03:37:58.083904 16154 tablet_replica.cc:330] T 5e409bda14f142959c8127f3bd2643fc P ec8c86cff51f48bf9ecc5177e6266b7b: stopping tablet replica
I20260807 03:37:58.084427 16154 raft_consensus.cc:2229] T 5e409bda14f142959c8127f3bd2643fc P ec8c86cff51f48bf9ecc5177e6266b7b [term 3 FOLLOWER]: Raft consensus shutting down.
I20260807 03:37:58.084836 16154 raft_consensus.cc:2258] T 5e409bda14f142959c8127f3bd2643fc P ec8c86cff51f48bf9ecc5177e6266b7b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 03:37:58.086377 16154 tablet_replica.cc:330] T a4e0d750de56427e8a0af5bc030271c4 P ec8c86cff51f48bf9ecc5177e6266b7b: stopping tablet replica
I20260807 03:37:58.086787 16154 raft_consensus.cc:2229] T a4e0d750de56427e8a0af5bc030271c4 P ec8c86cff51f48bf9ecc5177e6266b7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:37:58.087129 16154 raft_consensus.cc:2258] T a4e0d750de56427e8a0af5bc030271c4 P ec8c86cff51f48bf9ecc5177e6266b7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:37:58.088723 16154 tablet_replica.cc:330] T 485e8d9a337d4e4f9207f737e3a8812b P ec8c86cff51f48bf9ecc5177e6266b7b: stopping tablet replica
I20260807 03:37:58.089180 16154 raft_consensus.cc:2229] T 485e8d9a337d4e4f9207f737e3a8812b P ec8c86cff51f48bf9ecc5177e6266b7b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260807 03:37:58.089512 16154 raft_consensus.cc:2258] T 485e8d9a337d4e4f9207f737e3a8812b P ec8c86cff51f48bf9ecc5177e6266b7b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260807 03:37:58.090914 16154 tablet_replica.cc:330] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b: stopping tablet replica
I20260807 03:37:58.091289 16154 raft_consensus.cc:2229] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b [term 4 LEADER]: Raft consensus shutting down.
I20260807 03:37:58.091964 16154 raft_consensus.cc:2258] T fe184e38bf0d4e51bf42238d161ac738 P ec8c86cff51f48bf9ecc5177e6266b7b [term 4 FOLLOWER]: Raft consensus is shut down!
I20260807 03:37:58.119801 16154 tablet_server.cc:194] TabletServer@127.15.198.131:0 shutdown complete.
I20260807 03:37:58.136447 16154 master.cc:537] Master@127.15.198.190:41955 shutting down...
I20260807 03:37:58.531349 16154 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 044fc5f21a5d4b1597840dabbdca0f5a [term 1 LEADER]: Raft consensus shutting down.
I20260807 03:37:58.532011 16154 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 044fc5f21a5d4b1597840dabbdca0f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:37:58.532447 16154 tablet_replica.cc:330] T 00000000000000000000000000000000 P 044fc5f21a5d4b1597840dabbdca0f5a: stopping tablet replica
I20260807 03:37:58.550509 16154 master.cc:552] Master@127.15.198.190:41955 shutdown complete.
05aa1d36d28a48138040f6c78c4b34a5 4
9ca3bab29b4c4dce908de0247c715423 4
ec8c86cff51f48bf9ecc5177e6266b7b 1
05aa1d36d28a48138040f6c78c4b34a5 3
9ca3bab29b4c4dce908de0247c715423 3
ec8c86cff51f48bf9ecc5177e6266b7b 3
05aa1d36d28a48138040f6c78c4b34a5 8
9ca3bab29b4c4dce908de0247c715423 0
ec8c86cff51f48bf9ecc5177e6266b7b 1
05aa1d36d28a48138040f6c78c4b34a5 3
9ca3bab29b4c4dce908de0247c715423 3
ec8c86cff51f48bf9ecc5177e6266b7b 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (32722 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20260807 03:37:58.592656 16154 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.15.198.190:41401
I20260807 03:37:58.593704 16154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:37:58.599172 17040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:37:58.599448 17042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:37:58.600607 17044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:37:58.601967 16154 server_base.cc:1034] running on GCE node
I20260807 03:37:58.602993 16154 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260807 03:37:58.603178 16154 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260807 03:37:58.603365 16154 hybrid_clock.cc:648] HybridClock initialized: now 1786073878603336 us; error 0 us; skew 500 ppm
I20260807 03:37:58.603884 16154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:37:58.606712 16154 webserver.cc:460] Webserver started at http://127.15.198.190:37715/ using document root <none> and password file <none>
I20260807 03:37:58.607187 16154 fs_manager.cc:359] Metadata directory not provided
I20260807 03:37:58.607368 16154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:37:58.607676 16154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:37:58.608935 16154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786073845599630-16154-0/minicluster-data/master-0-root/instance:
uuid: "a2ac9ea50c6c4ec8a9d3d887fa5782c5"
format_stamp: "Formatted at 2026-08-07 03:37:58 on dist-test-slave-7vjv"
I20260807 03:37:58.613317 16154 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 03:37:58.616597 17054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:37:58.617273 16154 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260807 03:37:58.617532 16154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786073845599630-16154-0/minicluster-data/master-0-root
uuid: "a2ac9ea50c6c4ec8a9d3d887fa5782c5"
format_stamp: "Formatted at 2026-08-07 03:37:58 on dist-test-slave-7vjv"
I20260807 03:37:58.617787 16154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786073845599630-16154-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786073845599630-16154-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786073845599630-16154-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:37:58.629103 16154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:37:58.630288 16154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:37:58.668455 16154 rpc_server.cc:307] RPC server started. Bound to: 127.15.198.190:41401
I20260807 03:37:58.668560 17143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.198.190:41401 every 8 connection(s)
I20260807 03:37:58.672163 17144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:58.681298 17144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2ac9ea50c6c4ec8a9d3d887fa5782c5: Bootstrap starting.
I20260807 03:37:58.685513 17144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2ac9ea50c6c4ec8a9d3d887fa5782c5: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:58.689736 17144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2ac9ea50c6c4ec8a9d3d887fa5782c5: No bootstrap required, opened a new log
I20260807 03:37:58.691716 17144 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2ac9ea50c6c4ec8a9d3d887fa5782c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ac9ea50c6c4ec8a9d3d887fa5782c5" member_type: VOTER }
I20260807 03:37:58.692276 17144 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2ac9ea50c6c4ec8a9d3d887fa5782c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:58.692545 17144 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2ac9ea50c6c4ec8a9d3d887fa5782c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2ac9ea50c6c4ec8a9d3d887fa5782c5, State: Initialized, Role: FOLLOWER
I20260807 03:37:58.693267 17144 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2ac9ea50c6c4ec8a9d3d887fa5782c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ac9ea50c6c4ec8a9d3d887fa5782c5" member_type: VOTER }
I20260807 03:37:58.693706 17144 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2ac9ea50c6c4ec8a9d3d887fa5782c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:37:58.693890 17144 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2ac9ea50c6c4ec8a9d3d887fa5782c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:37:58.694109 17144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2ac9ea50c6c4ec8a9d3d887fa5782c5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:37:58.698072 17144 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2ac9ea50c6c4ec8a9d3d887fa5782c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ac9ea50c6c4ec8a9d3d887fa5782c5" member_type: VOTER }
I20260807 03:37:58.698518 17144 leader_election.cc:302] T 00000000000000000000000000000000 P a2ac9ea50c6c4ec8a9d3d887fa5782c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2ac9ea50c6c4ec8a9d3d887fa5782c5; no voters:
I20260807 03:37:58.699549 17144 leader_election.cc:288] T 00000000000000000000000000000000 P a2ac9ea50c6c4ec8a9d3d887fa5782c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:37:58.699990 17147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2ac9ea50c6c4ec8a9d3d887fa5782c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:37:58.701380 17147 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2ac9ea50c6c4ec8a9d3d887fa5782c5 [term 1 LEADER]: Becoming Leader. State: Replica: a2ac9ea50c6c4ec8a9d3d887fa5782c5, State: Running, Role: LEADER
I20260807 03:37:58.702062 17147 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2ac9ea50c6c4ec8a9d3d887fa5782c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ac9ea50c6c4ec8a9d3d887fa5782c5" member_type: VOTER }
I20260807 03:37:58.702577 17144 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2ac9ea50c6c4ec8a9d3d887fa5782c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:37:58.707801 17149 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2ac9ea50c6c4ec8a9d3d887fa5782c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2ac9ea50c6c4ec8a9d3d887fa5782c5. Latest consensus state: current_term: 1 leader_uuid: "a2ac9ea50c6c4ec8a9d3d887fa5782c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ac9ea50c6c4ec8a9d3d887fa5782c5" member_type: VOTER } }
I20260807 03:37:58.708496 17149 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2ac9ea50c6c4ec8a9d3d887fa5782c5 [sys.catalog]: This master's current role is: LEADER
I20260807 03:37:58.708446 17148 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2ac9ea50c6c4ec8a9d3d887fa5782c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2ac9ea50c6c4ec8a9d3d887fa5782c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ac9ea50c6c4ec8a9d3d887fa5782c5" member_type: VOTER } }
I20260807 03:37:58.709076 17148 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2ac9ea50c6c4ec8a9d3d887fa5782c5 [sys.catalog]: This master's current role is: LEADER
I20260807 03:37:58.711099 17156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:37:58.719005 17156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:37:58.720124 16154 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260807 03:37:58.727484 17156 catalog_manager.cc:1347] Generated new cluster ID: a3282234f0a245d1b052f542af1ae120
I20260807 03:37:58.727726 17156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:37:58.747665 17156 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:37:58.749203 17156 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:37:58.760737 17156 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2ac9ea50c6c4ec8a9d3d887fa5782c5: Generated new TSK 0
I20260807 03:37:58.761305 17156 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:37:58.786861 16154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:37:58.792719 17170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:37:58.793728 17172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:37:58.796890 17174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:37:58.797008 16154 server_base.cc:1034] running on GCE node
I20260807 03:37:58.798492 16154 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260807 03:37:58.798704 16154 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260807 03:37:58.798880 16154 hybrid_clock.cc:648] HybridClock initialized: now 1786073878798866 us; error 0 us; skew 500 ppm
I20260807 03:37:58.799401 16154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:37:58.801745 16154 webserver.cc:460] Webserver started at http://127.15.198.129:38397/ using document root <none> and password file <none>
I20260807 03:37:58.802249 16154 fs_manager.cc:359] Metadata directory not provided
I20260807 03:37:58.802451 16154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:37:58.802728 16154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:37:58.803829 16154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786073845599630-16154-0/minicluster-data/ts-0-root/instance:
uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
format_stamp: "Formatted at 2026-08-07 03:37:58 on dist-test-slave-7vjv"
I20260807 03:37:58.808424 16154 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260807 03:37:58.811491 17180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:37:58.812288 16154 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260807 03:37:58.812583 16154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786073845599630-16154-0/minicluster-data/ts-0-root
uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
format_stamp: "Formatted at 2026-08-07 03:37:58 on dist-test-slave-7vjv"
I20260807 03:37:58.812868 16154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786073845599630-16154-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786073845599630-16154-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786073845599630-16154-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:37:58.836964 16154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:37:58.838186 16154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:37:58.839636 16154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:37:58.841895 16154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:37:58.842077 16154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:37:58.842356 16154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:37:58.842532 16154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:37:58.882126 16154 rpc_server.cc:307] RPC server started. Bound to: 127.15.198.129:34887
I20260807 03:37:58.882221 17284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.198.129:34887 every 8 connection(s)
I20260807 03:37:58.887044 16154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:37:58.894457 17296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:37:58.897307 17298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:37:58.897784 17301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:37:58.899163 16154 server_base.cc:1034] running on GCE node
I20260807 03:37:58.900547 16154 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260807 03:37:58.900825 16154 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260807 03:37:58.900988 16154 hybrid_clock.cc:648] HybridClock initialized: now 1786073878900977 us; error 0 us; skew 500 ppm
I20260807 03:37:58.901476 17285 heartbeater.cc:347] Connected to a master server at 127.15.198.190:41401
I20260807 03:37:58.901583 16154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:37:58.901854 17285 heartbeater.cc:464] Registering TS with master...
I20260807 03:37:58.902661 17285 heartbeater.cc:511] Master 127.15.198.190:41401 requested a full tablet report, sending...
I20260807 03:37:58.904264 16154 webserver.cc:460] Webserver started at http://127.15.198.130:39167/ using document root <none> and password file <none>
I20260807 03:37:58.904803 16154 fs_manager.cc:359] Metadata directory not provided
I20260807 03:37:58.904817 17082 ts_manager.cc:194] Registered new tserver with Master: 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887)
I20260807 03:37:58.905087 16154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:37:58.905473 16154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:37:58.906575 16154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786073845599630-16154-0/minicluster-data/ts-1-root/instance:
uuid: "3cda358d5d184a4383a9716334390022"
format_stamp: "Formatted at 2026-08-07 03:37:58 on dist-test-slave-7vjv"
I20260807 03:37:58.906924 17082 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44268
I20260807 03:37:58.911080 16154 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 03:37:58.914502 17311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:37:58.915290 16154 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 03:37:58.915566 16154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786073845599630-16154-0/minicluster-data/ts-1-root
uuid: "3cda358d5d184a4383a9716334390022"
format_stamp: "Formatted at 2026-08-07 03:37:58 on dist-test-slave-7vjv"
I20260807 03:37:58.915853 16154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786073845599630-16154-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786073845599630-16154-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786073845599630-16154-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:37:58.936764 16154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:37:58.937964 16154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:37:58.939383 16154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:37:58.941638 16154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:37:58.941815 16154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:37:58.942104 16154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:37:58.942266 16154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:37:58.980041 16154 rpc_server.cc:307] RPC server started. Bound to: 127.15.198.130:45465
I20260807 03:37:58.980206 17415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.198.130:45465 every 8 connection(s)
I20260807 03:37:58.984647 16154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:37:58.991611 17424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:37:58.992583 17425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:37:58.995149 16154 server_base.cc:1034] running on GCE node
W20260807 03:37:58.996419 17428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:37:58.997529 16154 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260807 03:37:58.997768 16154 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260807 03:37:58.997884 16154 hybrid_clock.cc:648] HybridClock initialized: now 1786073878997876 us; error 0 us; skew 500 ppm
I20260807 03:37:58.998399 16154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:37:58.998528 17420 heartbeater.cc:347] Connected to a master server at 127.15.198.190:41401
I20260807 03:37:58.998940 17420 heartbeater.cc:464] Registering TS with master...
I20260807 03:37:58.999837 17420 heartbeater.cc:511] Master 127.15.198.190:41401 requested a full tablet report, sending...
I20260807 03:37:59.001327 16154 webserver.cc:460] Webserver started at http://127.15.198.131:40177/ using document root <none> and password file <none>
I20260807 03:37:59.001849 16154 fs_manager.cc:359] Metadata directory not provided
I20260807 03:37:59.001799 17082 ts_manager.cc:194] Registered new tserver with Master: 3cda358d5d184a4383a9716334390022 (127.15.198.130:45465)
I20260807 03:37:59.002239 16154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:37:59.002542 16154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:37:59.003856 17082 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44282
I20260807 03:37:59.003979 16154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786073845599630-16154-0/minicluster-data/ts-2-root/instance:
uuid: "2224f937b1b347e2931b7383c84894f2"
format_stamp: "Formatted at 2026-08-07 03:37:58 on dist-test-slave-7vjv"
I20260807 03:37:59.008510 16154 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 03:37:59.011735 17438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:37:59.012533 16154 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260807 03:37:59.012804 16154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786073845599630-16154-0/minicluster-data/ts-2-root
uuid: "2224f937b1b347e2931b7383c84894f2"
format_stamp: "Formatted at 2026-08-07 03:37:58 on dist-test-slave-7vjv"
I20260807 03:37:59.013129 16154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786073845599630-16154-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786073845599630-16154-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786073845599630-16154-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:37:59.028844 16154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:37:59.030037 16154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:37:59.031504 16154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:37:59.033758 16154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:37:59.033943 16154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:37:59.034109 16154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:37:59.034221 16154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:37:59.073232 16154 rpc_server.cc:307] RPC server started. Bound to: 127.15.198.131:41195
I20260807 03:37:59.073338 17539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.198.131:41195 every 8 connection(s)
I20260807 03:37:59.087918 17541 heartbeater.cc:347] Connected to a master server at 127.15.198.190:41401
I20260807 03:37:59.088368 17541 heartbeater.cc:464] Registering TS with master...
I20260807 03:37:59.089013 17541 heartbeater.cc:511] Master 127.15.198.190:41401 requested a full tablet report, sending...
I20260807 03:37:59.090728 17082 ts_manager.cc:194] Registered new tserver with Master: 2224f937b1b347e2931b7383c84894f2 (127.15.198.131:41195)
I20260807 03:37:59.090998 16154 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014519367s
I20260807 03:37:59.092370 17082 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44292
I20260807 03:37:59.093493 16154 test_util.cc:268] Using random seed: -1155850015
I20260807 03:37:59.116313 17082 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44300:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 03:37:59.119328 17082 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 03:37:59.193717 17481 tablet_service.cc:1467] Processing CreateTablet for tablet 96451ce78a6f45479178c8d596430538 (DEFAULT_TABLE table=test-workload [id=35552d16e8b143b1a0d1492e85dfe605]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260807 03:37:59.195231 17481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96451ce78a6f45479178c8d596430538. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:59.195699 17480 tablet_service.cc:1467] Processing CreateTablet for tablet e3f7b02979d648108ef56a2acf13fbf0 (DEFAULT_TABLE table=test-workload [id=35552d16e8b143b1a0d1492e85dfe605]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260807 03:37:59.196995 17480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3f7b02979d648108ef56a2acf13fbf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:59.203166 17479 tablet_service.cc:1467] Processing CreateTablet for tablet f99d56107b6a48db9611f1ce8282d575 (DEFAULT_TABLE table=test-workload [id=35552d16e8b143b1a0d1492e85dfe605]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260807 03:37:59.204463 17479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f99d56107b6a48db9611f1ce8282d575. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:59.207026 17475 tablet_service.cc:1467] Processing CreateTablet for tablet 8f6f16d016d3437d8aeae4a64c42a856 (DEFAULT_TABLE table=test-workload [id=35552d16e8b143b1a0d1492e85dfe605]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260807 03:37:59.207957 17466 tablet_service.cc:1467] Processing CreateTablet for tablet c93d1700ec3248b3a169ff40a4186010 (DEFAULT_TABLE table=test-workload [id=35552d16e8b143b1a0d1492e85dfe605]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260807 03:37:59.208376 17475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f6f16d016d3437d8aeae4a64c42a856. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:59.210525 17468 tablet_service.cc:1467] Processing CreateTablet for tablet 6f881ad9daec4be7b7ef3b0475c45474 (DEFAULT_TABLE table=test-workload [id=35552d16e8b143b1a0d1492e85dfe605]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260807 03:37:59.211155 17472 tablet_service.cc:1467] Processing CreateTablet for tablet 80b452de6935470789989a31d5305c47 (DEFAULT_TABLE table=test-workload [id=35552d16e8b143b1a0d1492e85dfe605]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260807 03:37:59.211423 17470 tablet_service.cc:1467] Processing CreateTablet for tablet 64fc83d22ad945ad890e67f9763fe413 (DEFAULT_TABLE table=test-workload [id=35552d16e8b143b1a0d1492e85dfe605]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260807 03:37:59.212172 17474 tablet_service.cc:1467] Processing CreateTablet for tablet a41a7b13d3284dcd8b9b616eeb93c578 (DEFAULT_TABLE table=test-workload [id=35552d16e8b143b1a0d1492e85dfe605]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260807 03:37:59.213577 17471 tablet_service.cc:1467] Processing CreateTablet for tablet efcd19aa11284c08ab3b4e1854a17dd9 (DEFAULT_TABLE table=test-workload [id=35552d16e8b143b1a0d1492e85dfe605]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260807 03:37:59.216457 17229 tablet_service.cc:1467] Processing CreateTablet for tablet e3f7b02979d648108ef56a2acf13fbf0 (DEFAULT_TABLE table=test-workload [id=35552d16e8b143b1a0d1492e85dfe605]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260807 03:37:59.217281 17226 tablet_service.cc:1467] Processing CreateTablet for tablet 8f6f16d016d3437d8aeae4a64c42a856 (DEFAULT_TABLE table=test-workload [id=35552d16e8b143b1a0d1492e85dfe605]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260807 03:37:59.217814 17229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3f7b02979d648108ef56a2acf13fbf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:59.218106 17209 tablet_service.cc:1467] Processing CreateTablet for tablet c93d1700ec3248b3a169ff40a4186010 (DEFAULT_TABLE table=test-workload [id=35552d16e8b143b1a0d1492e85dfe605]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260807 03:37:59.218597 17226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f6f16d016d3437d8aeae4a64c42a856. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:59.219337 17209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c93d1700ec3248b3a169ff40a4186010. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:59.221994 17224 tablet_service.cc:1467] Processing CreateTablet for tablet 80b452de6935470789989a31d5305c47 (DEFAULT_TABLE table=test-workload [id=35552d16e8b143b1a0d1492e85dfe605]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260807 03:37:59.222332 17216 tablet_service.cc:1467] Processing CreateTablet for tablet 6f881ad9daec4be7b7ef3b0475c45474 (DEFAULT_TABLE table=test-workload [id=35552d16e8b143b1a0d1492e85dfe605]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260807 03:37:59.215672 17228 tablet_service.cc:1467] Processing CreateTablet for tablet f99d56107b6a48db9611f1ce8282d575 (DEFAULT_TABLE table=test-workload [id=35552d16e8b143b1a0d1492e85dfe605]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260807 03:37:59.219942 17231 tablet_service.cc:1467] Processing CreateTablet for tablet 96451ce78a6f45479178c8d596430538 (DEFAULT_TABLE table=test-workload [id=35552d16e8b143b1a0d1492e85dfe605]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260807 03:37:59.225198 17223 tablet_service.cc:1467] Processing CreateTablet for tablet efcd19aa11284c08ab3b4e1854a17dd9 (DEFAULT_TABLE table=test-workload [id=35552d16e8b143b1a0d1492e85dfe605]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260807 03:37:59.226730 17225 tablet_service.cc:1467] Processing CreateTablet for tablet a41a7b13d3284dcd8b9b616eeb93c578 (DEFAULT_TABLE table=test-workload [id=35552d16e8b143b1a0d1492e85dfe605]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260807 03:37:59.227684 17466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c93d1700ec3248b3a169ff40a4186010. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:59.228284 17470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64fc83d22ad945ad890e67f9763fe413. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:59.227903 17217 tablet_service.cc:1467] Processing CreateTablet for tablet 64fc83d22ad945ad890e67f9763fe413 (DEFAULT_TABLE table=test-workload [id=35552d16e8b143b1a0d1492e85dfe605]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260807 03:37:59.229007 17472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80b452de6935470789989a31d5305c47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:59.229645 17474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a41a7b13d3284dcd8b9b616eeb93c578. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:59.230422 17468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f881ad9daec4be7b7ef3b0475c45474. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:59.230998 17471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efcd19aa11284c08ab3b4e1854a17dd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:59.247994 17224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80b452de6935470789989a31d5305c47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:59.252519 17223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efcd19aa11284c08ab3b4e1854a17dd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:59.253247 17217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64fc83d22ad945ad890e67f9763fe413. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:59.256553 17225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a41a7b13d3284dcd8b9b616eeb93c578. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:59.258296 17216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f881ad9daec4be7b7ef3b0475c45474. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:59.259112 17231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96451ce78a6f45479178c8d596430538. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:59.260421 17228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f99d56107b6a48db9611f1ce8282d575. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:59.281340 17349 tablet_service.cc:1467] Processing CreateTablet for tablet 96451ce78a6f45479178c8d596430538 (DEFAULT_TABLE table=test-workload [id=35552d16e8b143b1a0d1492e85dfe605]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260807 03:37:59.282758 17349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96451ce78a6f45479178c8d596430538. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:59.282517 17345 tablet_service.cc:1467] Processing CreateTablet for tablet 8f6f16d016d3437d8aeae4a64c42a856 (DEFAULT_TABLE table=test-workload [id=35552d16e8b143b1a0d1492e85dfe605]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260807 03:37:59.283830 17345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f6f16d016d3437d8aeae4a64c42a856. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:59.285005 17348 tablet_service.cc:1467] Processing CreateTablet for tablet e3f7b02979d648108ef56a2acf13fbf0 (DEFAULT_TABLE table=test-workload [id=35552d16e8b143b1a0d1492e85dfe605]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260807 03:37:59.286324 17348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3f7b02979d648108ef56a2acf13fbf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:59.289351 17346 tablet_service.cc:1467] Processing CreateTablet for tablet f99d56107b6a48db9611f1ce8282d575 (DEFAULT_TABLE table=test-workload [id=35552d16e8b143b1a0d1492e85dfe605]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260807 03:37:59.290692 17346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f99d56107b6a48db9611f1ce8282d575. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:59.295077 17343 tablet_service.cc:1467] Processing CreateTablet for tablet a41a7b13d3284dcd8b9b616eeb93c578 (DEFAULT_TABLE table=test-workload [id=35552d16e8b143b1a0d1492e85dfe605]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260807 03:37:59.296353 17343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a41a7b13d3284dcd8b9b616eeb93c578. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:59.304934 17563 tablet_bootstrap.cc:492] T 96451ce78a6f45479178c8d596430538 P 2224f937b1b347e2931b7383c84894f2: Bootstrap starting.
I20260807 03:37:59.309353 17342 tablet_service.cc:1467] Processing CreateTablet for tablet 80b452de6935470789989a31d5305c47 (DEFAULT_TABLE table=test-workload [id=35552d16e8b143b1a0d1492e85dfe605]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260807 03:37:59.312706 17342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80b452de6935470789989a31d5305c47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:59.325601 17340 tablet_service.cc:1467] Processing CreateTablet for tablet efcd19aa11284c08ab3b4e1854a17dd9 (DEFAULT_TABLE table=test-workload [id=35552d16e8b143b1a0d1492e85dfe605]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260807 03:37:59.326872 17340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efcd19aa11284c08ab3b4e1854a17dd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:59.329488 17348 tablet_service.cc:1467] Processing CreateTablet for tablet 64fc83d22ad945ad890e67f9763fe413 (DEFAULT_TABLE table=test-workload [id=35552d16e8b143b1a0d1492e85dfe605]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260807 03:37:59.329946 17339 tablet_service.cc:1467] Processing CreateTablet for tablet 6f881ad9daec4be7b7ef3b0475c45474 (DEFAULT_TABLE table=test-workload [id=35552d16e8b143b1a0d1492e85dfe605]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260807 03:37:59.330930 17348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64fc83d22ad945ad890e67f9763fe413. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:59.331542 17339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f881ad9daec4be7b7ef3b0475c45474. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:59.349279 17564 tablet_bootstrap.cc:492] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676: Bootstrap starting.
I20260807 03:37:59.353799 17338 tablet_service.cc:1467] Processing CreateTablet for tablet c93d1700ec3248b3a169ff40a4186010 (DEFAULT_TABLE table=test-workload [id=35552d16e8b143b1a0d1492e85dfe605]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260807 03:37:59.355121 17338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c93d1700ec3248b3a169ff40a4186010. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:37:59.359684 17563 tablet_bootstrap.cc:654] T 96451ce78a6f45479178c8d596430538 P 2224f937b1b347e2931b7383c84894f2: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:59.381114 17564 tablet_bootstrap.cc:654] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:59.385694 17565 tablet_bootstrap.cc:492] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022: Bootstrap starting.
I20260807 03:37:59.397754 17563 tablet_bootstrap.cc:492] T 96451ce78a6f45479178c8d596430538 P 2224f937b1b347e2931b7383c84894f2: No bootstrap required, opened a new log
I20260807 03:37:59.398191 17563 ts_tablet_manager.cc:1397] T 96451ce78a6f45479178c8d596430538 P 2224f937b1b347e2931b7383c84894f2: Time spent bootstrapping tablet: real 0.094s user 0.017s sys 0.015s
I20260807 03:37:59.398469 17565 tablet_bootstrap.cc:654] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:59.400969 17563 raft_consensus.cc:348] T 96451ce78a6f45479178c8d596430538 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } }
I20260807 03:37:59.401607 17563 raft_consensus.cc:374] T 96451ce78a6f45479178c8d596430538 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:59.401918 17563 raft_consensus.cc:729] T 96451ce78a6f45479178c8d596430538 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2224f937b1b347e2931b7383c84894f2, State: Initialized, Role: FOLLOWER
I20260807 03:37:59.402209 17564 tablet_bootstrap.cc:492] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676: No bootstrap required, opened a new log
I20260807 03:37:59.402559 17564 ts_tablet_manager.cc:1397] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676: Time spent bootstrapping tablet: real 0.054s user 0.009s sys 0.013s
I20260807 03:37:59.402716 17563 consensus_queue.cc:260] T 96451ce78a6f45479178c8d596430538 P 2224f937b1b347e2931b7383c84894f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } }
I20260807 03:37:59.404575 17565 tablet_bootstrap.cc:492] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022: No bootstrap required, opened a new log
I20260807 03:37:59.404989 17565 ts_tablet_manager.cc:1397] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.000s
I20260807 03:37:59.405258 17541 heartbeater.cc:503] Master 127.15.198.190:41401 was elected leader, sending a full tablet report...
I20260807 03:37:59.405699 17563 ts_tablet_manager.cc:1428] T 96451ce78a6f45479178c8d596430538 P 2224f937b1b347e2931b7383c84894f2: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260807 03:37:59.405244 17564 raft_consensus.cc:348] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.406049 17564 raft_consensus.cc:374] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:59.406323 17564 raft_consensus.cc:729] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c5dca27966a4b8cadb7cc0a8e0d1676, State: Initialized, Role: FOLLOWER
I20260807 03:37:59.406558 17563 tablet_bootstrap.cc:492] T 64fc83d22ad945ad890e67f9763fe413 P 2224f937b1b347e2931b7383c84894f2: Bootstrap starting.
I20260807 03:37:59.407037 17564 consensus_queue.cc:260] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.407531 17565 raft_consensus.cc:348] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } }
I20260807 03:37:59.408293 17565 raft_consensus.cc:374] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:59.408663 17565 raft_consensus.cc:729] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cda358d5d184a4383a9716334390022, State: Initialized, Role: FOLLOWER
I20260807 03:37:59.409701 17565 consensus_queue.cc:260] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } }
I20260807 03:37:59.412967 17563 tablet_bootstrap.cc:654] T 64fc83d22ad945ad890e67f9763fe413 P 2224f937b1b347e2931b7383c84894f2: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:59.424661 17563 tablet_bootstrap.cc:492] T 64fc83d22ad945ad890e67f9763fe413 P 2224f937b1b347e2931b7383c84894f2: No bootstrap required, opened a new log
I20260807 03:37:59.425125 17563 ts_tablet_manager.cc:1397] T 64fc83d22ad945ad890e67f9763fe413 P 2224f937b1b347e2931b7383c84894f2: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.006s
I20260807 03:37:59.427798 17563 raft_consensus.cc:348] T 64fc83d22ad945ad890e67f9763fe413 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.428598 17563 raft_consensus.cc:374] T 64fc83d22ad945ad890e67f9763fe413 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:59.428916 17563 raft_consensus.cc:729] T 64fc83d22ad945ad890e67f9763fe413 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2224f937b1b347e2931b7383c84894f2, State: Initialized, Role: FOLLOWER
I20260807 03:37:59.429826 17563 consensus_queue.cc:260] T 64fc83d22ad945ad890e67f9763fe413 P 2224f937b1b347e2931b7383c84894f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.432132 17563 ts_tablet_manager.cc:1428] T 64fc83d22ad945ad890e67f9763fe413 P 2224f937b1b347e2931b7383c84894f2: Time spent starting tablet: real 0.007s user 0.004s sys 0.001s
I20260807 03:37:59.433058 17563 tablet_bootstrap.cc:492] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2: Bootstrap starting.
I20260807 03:37:59.438228 17285 heartbeater.cc:503] Master 127.15.198.190:41401 was elected leader, sending a full tablet report...
I20260807 03:37:59.439293 17420 heartbeater.cc:503] Master 127.15.198.190:41401 was elected leader, sending a full tablet report...
I20260807 03:37:59.439633 17565 ts_tablet_manager.cc:1428] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022: Time spent starting tablet: real 0.034s user 0.011s sys 0.012s
I20260807 03:37:59.440549 17565 tablet_bootstrap.cc:492] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022: Bootstrap starting.
I20260807 03:37:59.445690 17564 ts_tablet_manager.cc:1428] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676: Time spent starting tablet: real 0.043s user 0.011s sys 0.025s
I20260807 03:37:59.446329 17565 tablet_bootstrap.cc:654] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:59.446687 17564 tablet_bootstrap.cc:492] T 6f881ad9daec4be7b7ef3b0475c45474 P 6c5dca27966a4b8cadb7cc0a8e0d1676: Bootstrap starting.
I20260807 03:37:59.448798 17574 raft_consensus.cc:482] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:37:59.449395 17574 raft_consensus.cc:504] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } }
I20260807 03:37:59.453972 17564 tablet_bootstrap.cc:654] T 6f881ad9daec4be7b7ef3b0475c45474 P 6c5dca27966a4b8cadb7cc0a8e0d1676: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:59.461378 17574 leader_election.cc:288] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2224f937b1b347e2931b7383c84894f2 (127.15.198.131:41195), 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887)
I20260807 03:37:59.466912 17564 tablet_bootstrap.cc:492] T 6f881ad9daec4be7b7ef3b0475c45474 P 6c5dca27966a4b8cadb7cc0a8e0d1676: No bootstrap required, opened a new log
I20260807 03:37:59.467512 17564 ts_tablet_manager.cc:1397] T 6f881ad9daec4be7b7ef3b0475c45474 P 6c5dca27966a4b8cadb7cc0a8e0d1676: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.006s
I20260807 03:37:59.469161 17496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96451ce78a6f45479178c8d596430538" candidate_uuid: "3cda358d5d184a4383a9716334390022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2224f937b1b347e2931b7383c84894f2" is_pre_election: true
I20260807 03:37:59.470027 17496 raft_consensus.cc:2454] T 96451ce78a6f45479178c8d596430538 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cda358d5d184a4383a9716334390022 in term 0.
I20260807 03:37:59.471354 17316 leader_election.cc:302] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2224f937b1b347e2931b7383c84894f2, 3cda358d5d184a4383a9716334390022; no voters:
I20260807 03:37:59.470983 17564 raft_consensus.cc:348] T 6f881ad9daec4be7b7ef3b0475c45474 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.472029 17564 raft_consensus.cc:374] T 6f881ad9daec4be7b7ef3b0475c45474 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:59.472445 17564 raft_consensus.cc:729] T 6f881ad9daec4be7b7ef3b0475c45474 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c5dca27966a4b8cadb7cc0a8e0d1676, State: Initialized, Role: FOLLOWER
I20260807 03:37:59.473421 17564 consensus_queue.cc:260] T 6f881ad9daec4be7b7ef3b0475c45474 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.475236 17574 raft_consensus.cc:2781] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:37:59.475616 17574 raft_consensus.cc:482] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:37:59.475934 17574 raft_consensus.cc:3037] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:37:59.476509 17564 ts_tablet_manager.cc:1428] T 6f881ad9daec4be7b7ef3b0475c45474 P 6c5dca27966a4b8cadb7cc0a8e0d1676: Time spent starting tablet: real 0.009s user 0.010s sys 0.001s
I20260807 03:37:59.477720 17564 tablet_bootstrap.cc:492] T 8f6f16d016d3437d8aeae4a64c42a856 P 6c5dca27966a4b8cadb7cc0a8e0d1676: Bootstrap starting.
I20260807 03:37:59.480485 17572 raft_consensus.cc:482] T 64fc83d22ad945ad890e67f9763fe413 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:37:59.480932 17572 raft_consensus.cc:504] T 64fc83d22ad945ad890e67f9763fe413 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.482053 17564 tablet_bootstrap.cc:654] T 8f6f16d016d3437d8aeae4a64c42a856 P 6c5dca27966a4b8cadb7cc0a8e0d1676: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:59.476665 17563 tablet_bootstrap.cc:654] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:59.482808 17574 raft_consensus.cc:504] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } }
I20260807 03:37:59.483788 17573 raft_consensus.cc:482] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:37:59.484347 17573 raft_consensus.cc:504] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.485077 17574 leader_election.cc:288] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022 [CANDIDATE]: Term 1 election: Requested vote from peers 2224f937b1b347e2931b7383c84894f2 (127.15.198.131:41195), 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887)
I20260807 03:37:59.486706 17573 leader_election.cc:288] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cda358d5d184a4383a9716334390022 (127.15.198.130:45465), 2224f937b1b347e2931b7383c84894f2 (127.15.198.131:41195)
W20260807 03:37:59.487378 17422 tablet.cc:2369] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:37:59.490039 17565 tablet_bootstrap.cc:492] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022: No bootstrap required, opened a new log
I20260807 03:37:59.491935 17565 ts_tablet_manager.cc:1397] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022: Time spent bootstrapping tablet: real 0.052s user 0.018s sys 0.020s
I20260807 03:37:59.495441 17565 raft_consensus.cc:348] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.496845 17565 raft_consensus.cc:374] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:59.497641 17565 raft_consensus.cc:729] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cda358d5d184a4383a9716334390022, State: Initialized, Role: FOLLOWER
I20260807 03:37:59.498890 17565 consensus_queue.cc:260] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.502601 17565 ts_tablet_manager.cc:1428] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022: Time spent starting tablet: real 0.010s user 0.003s sys 0.003s
I20260807 03:37:59.500620 17572 leader_election.cc:288] T 64fc83d22ad945ad890e67f9763fe413 P 2224f937b1b347e2931b7383c84894f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887), 3cda358d5d184a4383a9716334390022 (127.15.198.130:45465)
I20260807 03:37:59.505746 17565 tablet_bootstrap.cc:492] T 6f881ad9daec4be7b7ef3b0475c45474 P 3cda358d5d184a4383a9716334390022: Bootstrap starting.
I20260807 03:37:59.517782 17367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64fc83d22ad945ad890e67f9763fe413" candidate_uuid: "2224f937b1b347e2931b7383c84894f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cda358d5d184a4383a9716334390022" is_pre_election: true
W20260807 03:37:59.520435 17443 leader_election.cc:341] T 64fc83d22ad945ad890e67f9763fe413 P 2224f937b1b347e2931b7383c84894f2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3cda358d5d184a4383a9716334390022 (127.15.198.130:45465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:37:59.520778 17565 tablet_bootstrap.cc:654] T 6f881ad9daec4be7b7ef3b0475c45474 P 3cda358d5d184a4383a9716334390022: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:59.535480 17565 tablet_bootstrap.cc:492] T 6f881ad9daec4be7b7ef3b0475c45474 P 3cda358d5d184a4383a9716334390022: No bootstrap required, opened a new log
I20260807 03:37:59.535297 17247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96451ce78a6f45479178c8d596430538" candidate_uuid: "3cda358d5d184a4383a9716334390022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" is_pre_election: true
I20260807 03:37:59.536485 17246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96451ce78a6f45479178c8d596430538" candidate_uuid: "3cda358d5d184a4383a9716334390022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
W20260807 03:37:59.539383 17313 leader_election.cc:341] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:37:59.539906 17496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96451ce78a6f45479178c8d596430538" candidate_uuid: "3cda358d5d184a4383a9716334390022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2224f937b1b347e2931b7383c84894f2"
I20260807 03:37:59.540364 17564 tablet_bootstrap.cc:492] T 8f6f16d016d3437d8aeae4a64c42a856 P 6c5dca27966a4b8cadb7cc0a8e0d1676: No bootstrap required, opened a new log
W20260807 03:37:59.540297 17313 leader_election.cc:341] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:37:59.540503 17496 raft_consensus.cc:3037] T 96451ce78a6f45479178c8d596430538 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:37:59.540920 17564 ts_tablet_manager.cc:1397] T 8f6f16d016d3437d8aeae4a64c42a856 P 6c5dca27966a4b8cadb7cc0a8e0d1676: Time spent bootstrapping tablet: real 0.063s user 0.009s sys 0.018s
I20260807 03:37:59.541311 17565 ts_tablet_manager.cc:1397] T 6f881ad9daec4be7b7ef3b0475c45474 P 3cda358d5d184a4383a9716334390022: Time spent bootstrapping tablet: real 0.036s user 0.016s sys 0.000s
I20260807 03:37:59.543434 17564 raft_consensus.cc:348] T 8f6f16d016d3437d8aeae4a64c42a856 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.544183 17564 raft_consensus.cc:374] T 8f6f16d016d3437d8aeae4a64c42a856 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:59.544467 17564 raft_consensus.cc:729] T 8f6f16d016d3437d8aeae4a64c42a856 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c5dca27966a4b8cadb7cc0a8e0d1676, State: Initialized, Role: FOLLOWER
I20260807 03:37:59.545053 17563 tablet_bootstrap.cc:492] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2: No bootstrap required, opened a new log
I20260807 03:37:59.545845 17563 ts_tablet_manager.cc:1397] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2: Time spent bootstrapping tablet: real 0.113s user 0.018s sys 0.014s
I20260807 03:37:59.547398 17496 raft_consensus.cc:2454] T 96451ce78a6f45479178c8d596430538 P 2224f937b1b347e2931b7383c84894f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cda358d5d184a4383a9716334390022 in term 1.
I20260807 03:37:59.548480 17316 leader_election.cc:302] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2224f937b1b347e2931b7383c84894f2, 3cda358d5d184a4383a9716334390022; no voters: 6c5dca27966a4b8cadb7cc0a8e0d1676
I20260807 03:37:59.549280 17574 raft_consensus.cc:2781] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:37:59.550479 17574 raft_consensus.cc:686] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022 [term 1 LEADER]: Becoming Leader. State: Replica: 3cda358d5d184a4383a9716334390022, State: Running, Role: LEADER
I20260807 03:37:59.549401 17563 raft_consensus.cc:348] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.551077 17563 raft_consensus.cc:374] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:59.551440 17563 raft_consensus.cc:729] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2224f937b1b347e2931b7383c84894f2, State: Initialized, Role: FOLLOWER
I20260807 03:37:59.551352 17574 consensus_queue.cc:237] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } }
I20260807 03:37:59.552181 17564 consensus_queue.cc:260] T 8f6f16d016d3437d8aeae4a64c42a856 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.552508 17563 consensus_queue.cc:260] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.554764 17563 ts_tablet_manager.cc:1428] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20260807 03:37:59.550197 17565 raft_consensus.cc:348] T 6f881ad9daec4be7b7ef3b0475c45474 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.555414 17573 raft_consensus.cc:482] T 8f6f16d016d3437d8aeae4a64c42a856 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:37:59.555948 17573 raft_consensus.cc:504] T 8f6f16d016d3437d8aeae4a64c42a856 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.557019 17564 ts_tablet_manager.cc:1428] T 8f6f16d016d3437d8aeae4a64c42a856 P 6c5dca27966a4b8cadb7cc0a8e0d1676: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20260807 03:37:59.557834 17573 leader_election.cc:288] T 8f6f16d016d3437d8aeae4a64c42a856 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cda358d5d184a4383a9716334390022 (127.15.198.130:45465), 2224f937b1b347e2931b7383c84894f2 (127.15.198.131:41195)
I20260807 03:37:59.558560 17565 raft_consensus.cc:374] T 6f881ad9daec4be7b7ef3b0475c45474 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:59.558990 17565 raft_consensus.cc:729] T 6f881ad9daec4be7b7ef3b0475c45474 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cda358d5d184a4383a9716334390022, State: Initialized, Role: FOLLOWER
I20260807 03:37:59.557845 17564 tablet_bootstrap.cc:492] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676: Bootstrap starting.
I20260807 03:37:59.560037 17565 consensus_queue.cc:260] T 6f881ad9daec4be7b7ef3b0475c45474 P 3cda358d5d184a4383a9716334390022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.560858 17246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64fc83d22ad945ad890e67f9763fe413" candidate_uuid: "2224f937b1b347e2931b7383c84894f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" is_pre_election: true
W20260807 03:37:59.562273 17440 leader_election.cc:341] T 64fc83d22ad945ad890e67f9763fe413 P 2224f937b1b347e2931b7383c84894f2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:37:59.562816 17565 ts_tablet_manager.cc:1428] T 6f881ad9daec4be7b7ef3b0475c45474 P 3cda358d5d184a4383a9716334390022: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20260807 03:37:59.562707 17440 leader_election.cc:302] T 64fc83d22ad945ad890e67f9763fe413 P 2224f937b1b347e2931b7383c84894f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2224f937b1b347e2931b7383c84894f2; no voters: 3cda358d5d184a4383a9716334390022, 6c5dca27966a4b8cadb7cc0a8e0d1676
I20260807 03:37:59.563665 17563 tablet_bootstrap.cc:492] T f99d56107b6a48db9611f1ce8282d575 P 2224f937b1b347e2931b7383c84894f2: Bootstrap starting.
I20260807 03:37:59.567471 17572 raft_consensus.cc:2726] T 64fc83d22ad945ad890e67f9763fe413 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:37:59.568614 17572 raft_consensus.cc:482] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:37:59.569131 17572 raft_consensus.cc:504] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.564808 17565 tablet_bootstrap.cc:492] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022: Bootstrap starting.
I20260807 03:37:59.572844 17367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f7b02979d648108ef56a2acf13fbf0" candidate_uuid: "2224f937b1b347e2931b7383c84894f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cda358d5d184a4383a9716334390022" is_pre_election: true
I20260807 03:37:59.573648 17367 raft_consensus.cc:2454] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2224f937b1b347e2931b7383c84894f2 in term 0.
I20260807 03:37:59.575075 17443 leader_election.cc:302] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2224f937b1b347e2931b7383c84894f2, 3cda358d5d184a4383a9716334390022; no voters:
I20260807 03:37:59.584973 17565 tablet_bootstrap.cc:654] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:59.585244 17564 tablet_bootstrap.cc:654] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:59.589205 17563 tablet_bootstrap.cc:654] T f99d56107b6a48db9611f1ce8282d575 P 2224f937b1b347e2931b7383c84894f2: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:59.589289 17246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f7b02979d648108ef56a2acf13fbf0" candidate_uuid: "2224f937b1b347e2931b7383c84894f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" is_pre_election: true
I20260807 03:37:59.590570 17246 raft_consensus.cc:2454] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2224f937b1b347e2931b7383c84894f2 in term 0.
I20260807 03:37:59.576633 17572 leader_election.cc:288] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887), 3cda358d5d184a4383a9716334390022 (127.15.198.130:45465)
I20260807 03:37:59.591818 17572 raft_consensus.cc:2781] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:37:59.592680 17572 raft_consensus.cc:482] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:37:59.593106 17572 raft_consensus.cc:3037] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:37:59.597575 17082 catalog_manager.cc:5495] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cda358d5d184a4383a9716334390022 (127.15.198.130). New cstate: current_term: 1 leader_uuid: "3cda358d5d184a4383a9716334390022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } health_report { overall_health: UNKNOWN } } }
I20260807 03:37:59.604514 17564 tablet_bootstrap.cc:492] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676: No bootstrap required, opened a new log
I20260807 03:37:59.604964 17564 ts_tablet_manager.cc:1397] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676: Time spent bootstrapping tablet: real 0.047s user 0.011s sys 0.000s
I20260807 03:37:59.607596 17564 raft_consensus.cc:348] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.608309 17564 raft_consensus.cc:374] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:59.608628 17564 raft_consensus.cc:729] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c5dca27966a4b8cadb7cc0a8e0d1676, State: Initialized, Role: FOLLOWER
I20260807 03:37:59.612906 17564 consensus_queue.cc:260] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.616370 17563 tablet_bootstrap.cc:492] T f99d56107b6a48db9611f1ce8282d575 P 2224f937b1b347e2931b7383c84894f2: No bootstrap required, opened a new log
I20260807 03:37:59.616930 17563 ts_tablet_manager.cc:1397] T f99d56107b6a48db9611f1ce8282d575 P 2224f937b1b347e2931b7383c84894f2: Time spent bootstrapping tablet: real 0.053s user 0.013s sys 0.005s
I20260807 03:37:59.619181 17367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f7b02979d648108ef56a2acf13fbf0" candidate_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cda358d5d184a4383a9716334390022" is_pre_election: true
I20260807 03:37:59.619827 17367 raft_consensus.cc:2454] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c5dca27966a4b8cadb7cc0a8e0d1676 in term 0.
I20260807 03:37:59.619457 17563 raft_consensus.cc:348] T f99d56107b6a48db9611f1ce8282d575 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.620952 17366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f6f16d016d3437d8aeae4a64c42a856" candidate_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cda358d5d184a4383a9716334390022" is_pre_election: true
I20260807 03:37:59.620867 17572 raft_consensus.cc:504] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.621887 17565 tablet_bootstrap.cc:492] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022: No bootstrap required, opened a new log
I20260807 03:37:59.622323 17565 ts_tablet_manager.cc:1397] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022: Time spent bootstrapping tablet: real 0.058s user 0.009s sys 0.005s
I20260807 03:37:59.622759 17184 leader_election.cc:302] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cda358d5d184a4383a9716334390022, 6c5dca27966a4b8cadb7cc0a8e0d1676; no voters:
I20260807 03:37:59.623436 17246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f7b02979d648108ef56a2acf13fbf0" candidate_uuid: "2224f937b1b347e2931b7383c84894f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
I20260807 03:37:59.623608 17573 raft_consensus.cc:2781] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:37:59.624059 17246 raft_consensus.cc:3037] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Advancing to term 1
W20260807 03:37:59.624511 17184 leader_election.cc:341] T 8f6f16d016d3437d8aeae4a64c42a856 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3cda358d5d184a4383a9716334390022 (127.15.198.130:45465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:37:59.624958 17565 raft_consensus.cc:348] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.625711 17565 raft_consensus.cc:374] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:59.625988 17565 raft_consensus.cc:729] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cda358d5d184a4383a9716334390022, State: Initialized, Role: FOLLOWER
I20260807 03:37:59.620332 17563 raft_consensus.cc:374] T f99d56107b6a48db9611f1ce8282d575 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:59.626675 17563 raft_consensus.cc:729] T f99d56107b6a48db9611f1ce8282d575 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2224f937b1b347e2931b7383c84894f2, State: Initialized, Role: FOLLOWER
I20260807 03:37:59.627363 17563 consensus_queue.cc:260] T f99d56107b6a48db9611f1ce8282d575 P 2224f937b1b347e2931b7383c84894f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.629511 17563 ts_tablet_manager.cc:1428] T f99d56107b6a48db9611f1ce8282d575 P 2224f937b1b347e2931b7383c84894f2: Time spent starting tablet: real 0.012s user 0.003s sys 0.002s
I20260807 03:37:59.630355 17563 tablet_bootstrap.cc:492] T a41a7b13d3284dcd8b9b616eeb93c578 P 2224f937b1b347e2931b7383c84894f2: Bootstrap starting.
I20260807 03:37:59.631323 17564 ts_tablet_manager.cc:1428] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676: Time spent starting tablet: real 0.026s user 0.007s sys 0.000s
I20260807 03:37:59.631500 17246 raft_consensus.cc:2454] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2224f937b1b347e2931b7383c84894f2 in term 1.
I20260807 03:37:59.631938 17573 raft_consensus.cc:482] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:37:59.632292 17573 raft_consensus.cc:3037] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:37:59.632346 17565 consensus_queue.cc:260] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.633841 17564 tablet_bootstrap.cc:492] T f99d56107b6a48db9611f1ce8282d575 P 6c5dca27966a4b8cadb7cc0a8e0d1676: Bootstrap starting.
I20260807 03:37:59.634776 17565 ts_tablet_manager.cc:1428] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022: Time spent starting tablet: real 0.012s user 0.006s sys 0.001s
I20260807 03:37:59.635686 17565 tablet_bootstrap.cc:492] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022: Bootstrap starting.
I20260807 03:37:59.641876 17574 raft_consensus.cc:482] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:37:59.642215 17564 tablet_bootstrap.cc:654] T f99d56107b6a48db9611f1ce8282d575 P 6c5dca27966a4b8cadb7cc0a8e0d1676: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:59.642325 17574 raft_consensus.cc:504] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.644331 17574 leader_election.cc:288] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887), 2224f937b1b347e2931b7383c84894f2 (127.15.198.131:41195)
I20260807 03:37:59.642274 17496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f7b02979d648108ef56a2acf13fbf0" candidate_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2224f937b1b347e2931b7383c84894f2" is_pre_election: true
I20260807 03:37:59.645185 17496 raft_consensus.cc:2379] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6c5dca27966a4b8cadb7cc0a8e0d1676 in current term 1: Already voted for candidate 2224f937b1b347e2931b7383c84894f2 in this term.
I20260807 03:37:59.645210 17494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f6f16d016d3437d8aeae4a64c42a856" candidate_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2224f937b1b347e2931b7383c84894f2" is_pre_election: true
I20260807 03:37:59.646263 17247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b452de6935470789989a31d5305c47" candidate_uuid: "3cda358d5d184a4383a9716334390022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" is_pre_election: true
W20260807 03:37:59.647655 17184 leader_election.cc:341] T 8f6f16d016d3437d8aeae4a64c42a856 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2224f937b1b347e2931b7383c84894f2 (127.15.198.131:41195): Illegal state: must be running to vote when last-logged opid is not known
W20260807 03:37:59.647743 17313 leader_election.cc:341] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:37:59.648079 17184 leader_election.cc:302] T 8f6f16d016d3437d8aeae4a64c42a856 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c5dca27966a4b8cadb7cc0a8e0d1676; no voters: 2224f937b1b347e2931b7383c84894f2, 3cda358d5d184a4383a9716334390022
I20260807 03:37:59.648912 17494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b452de6935470789989a31d5305c47" candidate_uuid: "3cda358d5d184a4383a9716334390022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2224f937b1b347e2931b7383c84894f2" is_pre_election: true
W20260807 03:37:59.650115 17316 leader_election.cc:341] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2224f937b1b347e2931b7383c84894f2 (127.15.198.131:41195): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:37:59.650496 17316 leader_election.cc:302] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3cda358d5d184a4383a9716334390022; no voters: 2224f937b1b347e2931b7383c84894f2, 6c5dca27966a4b8cadb7cc0a8e0d1676
I20260807 03:37:59.651477 17574 raft_consensus.cc:2726] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:37:59.651911 17573 raft_consensus.cc:504] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.655159 17564 tablet_bootstrap.cc:492] T f99d56107b6a48db9611f1ce8282d575 P 6c5dca27966a4b8cadb7cc0a8e0d1676: No bootstrap required, opened a new log
I20260807 03:37:59.655328 17563 tablet_bootstrap.cc:654] T a41a7b13d3284dcd8b9b616eeb93c578 P 2224f937b1b347e2931b7383c84894f2: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:59.655613 17564 ts_tablet_manager.cc:1397] T f99d56107b6a48db9611f1ce8282d575 P 6c5dca27966a4b8cadb7cc0a8e0d1676: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.011s
I20260807 03:37:59.657461 17565 tablet_bootstrap.cc:654] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:59.658318 17573 leader_election.cc:288] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [CANDIDATE]: Term 2 election: Requested vote from peers 3cda358d5d184a4383a9716334390022 (127.15.198.130:45465), 2224f937b1b347e2931b7383c84894f2 (127.15.198.131:41195)
I20260807 03:37:59.658205 17564 raft_consensus.cc:348] T f99d56107b6a48db9611f1ce8282d575 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.659017 17564 raft_consensus.cc:374] T f99d56107b6a48db9611f1ce8282d575 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:59.659047 17573 raft_consensus.cc:2726] T 8f6f16d016d3437d8aeae4a64c42a856 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:37:59.659584 17564 raft_consensus.cc:729] T f99d56107b6a48db9611f1ce8282d575 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c5dca27966a4b8cadb7cc0a8e0d1676, State: Initialized, Role: FOLLOWER
I20260807 03:37:59.660418 17564 consensus_queue.cc:260] T f99d56107b6a48db9611f1ce8282d575 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.661077 17440 leader_election.cc:302] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2224f937b1b347e2931b7383c84894f2, 6c5dca27966a4b8cadb7cc0a8e0d1676; no voters:
I20260807 03:37:59.662202 17600 raft_consensus.cc:2781] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:37:59.662552 17572 leader_election.cc:288] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [CANDIDATE]: Term 1 election: Requested vote from peers 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887), 3cda358d5d184a4383a9716334390022 (127.15.198.130:45465)
I20260807 03:37:59.663182 17564 ts_tablet_manager.cc:1428] T f99d56107b6a48db9611f1ce8282d575 P 6c5dca27966a4b8cadb7cc0a8e0d1676: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260807 03:37:59.664350 17366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f7b02979d648108ef56a2acf13fbf0" candidate_uuid: "2224f937b1b347e2931b7383c84894f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cda358d5d184a4383a9716334390022"
I20260807 03:37:59.664921 17366 raft_consensus.cc:3037] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:37:59.672288 17367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f7b02979d648108ef56a2acf13fbf0" candidate_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cda358d5d184a4383a9716334390022"
I20260807 03:37:59.674422 17494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f7b02979d648108ef56a2acf13fbf0" candidate_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2224f937b1b347e2931b7383c84894f2"
I20260807 03:37:59.674901 17565 tablet_bootstrap.cc:492] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022: No bootstrap required, opened a new log
I20260807 03:37:59.675555 17565 ts_tablet_manager.cc:1397] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022: Time spent bootstrapping tablet: real 0.040s user 0.015s sys 0.007s
I20260807 03:37:59.677683 17366 raft_consensus.cc:2454] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2224f937b1b347e2931b7383c84894f2 in term 1.
I20260807 03:37:59.678254 17565 raft_consensus.cc:348] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.679029 17565 raft_consensus.cc:374] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:59.679217 17564 tablet_bootstrap.cc:492] T 64fc83d22ad945ad890e67f9763fe413 P 6c5dca27966a4b8cadb7cc0a8e0d1676: Bootstrap starting.
I20260807 03:37:59.679364 17565 raft_consensus.cc:729] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cda358d5d184a4383a9716334390022, State: Initialized, Role: FOLLOWER
I20260807 03:37:59.680320 17565 consensus_queue.cc:260] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.681910 17563 tablet_bootstrap.cc:492] T a41a7b13d3284dcd8b9b616eeb93c578 P 2224f937b1b347e2931b7383c84894f2: No bootstrap required, opened a new log
I20260807 03:37:59.682343 17563 ts_tablet_manager.cc:1397] T a41a7b13d3284dcd8b9b616eeb93c578 P 2224f937b1b347e2931b7383c84894f2: Time spent bootstrapping tablet: real 0.052s user 0.010s sys 0.003s
I20260807 03:37:59.682716 17565 ts_tablet_manager.cc:1428] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260807 03:37:59.683749 17565 tablet_bootstrap.cc:492] T 8f6f16d016d3437d8aeae4a64c42a856 P 3cda358d5d184a4383a9716334390022: Bootstrap starting.
I20260807 03:37:59.685344 17564 tablet_bootstrap.cc:654] T 64fc83d22ad945ad890e67f9763fe413 P 6c5dca27966a4b8cadb7cc0a8e0d1676: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:59.685141 17563 raft_consensus.cc:348] T a41a7b13d3284dcd8b9b616eeb93c578 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.685886 17563 raft_consensus.cc:374] T a41a7b13d3284dcd8b9b616eeb93c578 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:59.686189 17563 raft_consensus.cc:729] T a41a7b13d3284dcd8b9b616eeb93c578 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2224f937b1b347e2931b7383c84894f2, State: Initialized, Role: FOLLOWER
I20260807 03:37:59.687603 17563 consensus_queue.cc:260] T a41a7b13d3284dcd8b9b616eeb93c578 P 2224f937b1b347e2931b7383c84894f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.689791 17563 ts_tablet_manager.cc:1428] T a41a7b13d3284dcd8b9b616eeb93c578 P 2224f937b1b347e2931b7383c84894f2: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260807 03:37:59.690068 17600 raft_consensus.cc:686] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 1 LEADER]: Becoming Leader. State: Replica: 2224f937b1b347e2931b7383c84894f2, State: Running, Role: LEADER
I20260807 03:37:59.690654 17563 tablet_bootstrap.cc:492] T c93d1700ec3248b3a169ff40a4186010 P 2224f937b1b347e2931b7383c84894f2: Bootstrap starting.
I20260807 03:37:59.691215 17565 tablet_bootstrap.cc:654] T 8f6f16d016d3437d8aeae4a64c42a856 P 3cda358d5d184a4383a9716334390022: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:59.690902 17600 consensus_queue.cc:237] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.698228 17563 tablet_bootstrap.cc:654] T c93d1700ec3248b3a169ff40a4186010 P 2224f937b1b347e2931b7383c84894f2: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:59.701310 17564 tablet_bootstrap.cc:492] T 64fc83d22ad945ad890e67f9763fe413 P 6c5dca27966a4b8cadb7cc0a8e0d1676: No bootstrap required, opened a new log
I20260807 03:37:59.701807 17564 ts_tablet_manager.cc:1397] T 64fc83d22ad945ad890e67f9763fe413 P 6c5dca27966a4b8cadb7cc0a8e0d1676: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.008s
I20260807 03:37:59.703534 17565 tablet_bootstrap.cc:492] T 8f6f16d016d3437d8aeae4a64c42a856 P 3cda358d5d184a4383a9716334390022: No bootstrap required, opened a new log
I20260807 03:37:59.704046 17565 ts_tablet_manager.cc:1397] T 8f6f16d016d3437d8aeae4a64c42a856 P 3cda358d5d184a4383a9716334390022: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.007s
I20260807 03:37:59.704902 17564 raft_consensus.cc:348] T 64fc83d22ad945ad890e67f9763fe413 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.705710 17564 raft_consensus.cc:374] T 64fc83d22ad945ad890e67f9763fe413 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:59.705998 17564 raft_consensus.cc:729] T 64fc83d22ad945ad890e67f9763fe413 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c5dca27966a4b8cadb7cc0a8e0d1676, State: Initialized, Role: FOLLOWER
I20260807 03:37:59.706678 17564 consensus_queue.cc:260] T 64fc83d22ad945ad890e67f9763fe413 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.706873 17565 raft_consensus.cc:348] T 8f6f16d016d3437d8aeae4a64c42a856 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.708063 17184 leader_election.cc:302] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c5dca27966a4b8cadb7cc0a8e0d1676; no voters: 2224f937b1b347e2931b7383c84894f2, 3cda358d5d184a4383a9716334390022
I20260807 03:37:59.710137 17565 raft_consensus.cc:374] T 8f6f16d016d3437d8aeae4a64c42a856 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:59.710459 17607 raft_consensus.cc:2726] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 03:37:59.710497 17565 raft_consensus.cc:729] T 8f6f16d016d3437d8aeae4a64c42a856 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cda358d5d184a4383a9716334390022, State: Initialized, Role: FOLLOWER
I20260807 03:37:59.711688 17565 consensus_queue.cc:260] T 8f6f16d016d3437d8aeae4a64c42a856 P 3cda358d5d184a4383a9716334390022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.711725 17080 catalog_manager.cc:5495] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2224f937b1b347e2931b7383c84894f2 (127.15.198.131). New cstate: current_term: 1 leader_uuid: "2224f937b1b347e2931b7383c84894f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } health_report { overall_health: HEALTHY } } }
I20260807 03:37:59.714625 17565 ts_tablet_manager.cc:1428] T 8f6f16d016d3437d8aeae4a64c42a856 P 3cda358d5d184a4383a9716334390022: Time spent starting tablet: real 0.010s user 0.002s sys 0.003s
I20260807 03:37:59.715600 17565 tablet_bootstrap.cc:492] T f99d56107b6a48db9611f1ce8282d575 P 3cda358d5d184a4383a9716334390022: Bootstrap starting.
W20260807 03:37:59.719679 17159 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 03:37:59.721545 17565 tablet_bootstrap.cc:654] T f99d56107b6a48db9611f1ce8282d575 P 3cda358d5d184a4383a9716334390022: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:59.723011 17564 ts_tablet_manager.cc:1428] T 64fc83d22ad945ad890e67f9763fe413 P 6c5dca27966a4b8cadb7cc0a8e0d1676: Time spent starting tablet: real 0.021s user 0.007s sys 0.000s
I20260807 03:37:59.723696 17574 raft_consensus.cc:482] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:37:59.723951 17564 tablet_bootstrap.cc:492] T a41a7b13d3284dcd8b9b616eeb93c578 P 6c5dca27966a4b8cadb7cc0a8e0d1676: Bootstrap starting.
I20260807 03:37:59.724151 17574 raft_consensus.cc:504] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.726091 17574 leader_election.cc:288] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887), 2224f937b1b347e2931b7383c84894f2 (127.15.198.131:41195)
I20260807 03:37:59.730728 17564 tablet_bootstrap.cc:654] T a41a7b13d3284dcd8b9b616eeb93c578 P 6c5dca27966a4b8cadb7cc0a8e0d1676: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:59.736480 17494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a41a7b13d3284dcd8b9b616eeb93c578" candidate_uuid: "3cda358d5d184a4383a9716334390022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2224f937b1b347e2931b7383c84894f2" is_pre_election: true
I20260807 03:37:59.737145 17494 raft_consensus.cc:2454] T a41a7b13d3284dcd8b9b616eeb93c578 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cda358d5d184a4383a9716334390022 in term 0.
I20260807 03:37:59.739984 17316 leader_election.cc:302] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2224f937b1b347e2931b7383c84894f2, 3cda358d5d184a4383a9716334390022; no voters:
I20260807 03:37:59.740263 17246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a41a7b13d3284dcd8b9b616eeb93c578" candidate_uuid: "3cda358d5d184a4383a9716334390022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" is_pre_election: true
I20260807 03:37:59.740813 17574 raft_consensus.cc:2781] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:37:59.741137 17574 raft_consensus.cc:482] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 03:37:59.741349 17313 leader_election.cc:341] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:37:59.741443 17574 raft_consensus.cc:3037] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:37:59.747816 17563 tablet_bootstrap.cc:492] T c93d1700ec3248b3a169ff40a4186010 P 2224f937b1b347e2931b7383c84894f2: No bootstrap required, opened a new log
I20260807 03:37:59.748355 17563 ts_tablet_manager.cc:1397] T c93d1700ec3248b3a169ff40a4186010 P 2224f937b1b347e2931b7383c84894f2: Time spent bootstrapping tablet: real 0.058s user 0.016s sys 0.029s
I20260807 03:37:59.751087 17563 raft_consensus.cc:348] T c93d1700ec3248b3a169ff40a4186010 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.751215 17574 raft_consensus.cc:504] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.751861 17563 raft_consensus.cc:374] T c93d1700ec3248b3a169ff40a4186010 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:59.752360 17563 raft_consensus.cc:729] T c93d1700ec3248b3a169ff40a4186010 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2224f937b1b347e2931b7383c84894f2, State: Initialized, Role: FOLLOWER
I20260807 03:37:59.752689 17565 tablet_bootstrap.cc:492] T f99d56107b6a48db9611f1ce8282d575 P 3cda358d5d184a4383a9716334390022: No bootstrap required, opened a new log
I20260807 03:37:59.753106 17565 ts_tablet_manager.cc:1397] T f99d56107b6a48db9611f1ce8282d575 P 3cda358d5d184a4383a9716334390022: Time spent bootstrapping tablet: real 0.038s user 0.009s sys 0.016s
I20260807 03:37:59.753248 17563 consensus_queue.cc:260] T c93d1700ec3248b3a169ff40a4186010 P 2224f937b1b347e2931b7383c84894f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.754670 17246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a41a7b13d3284dcd8b9b616eeb93c578" candidate_uuid: "3cda358d5d184a4383a9716334390022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
I20260807 03:37:59.755628 17563 ts_tablet_manager.cc:1428] T c93d1700ec3248b3a169ff40a4186010 P 2224f937b1b347e2931b7383c84894f2: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260807 03:37:59.755643 17565 raft_consensus.cc:348] T f99d56107b6a48db9611f1ce8282d575 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.756448 17565 raft_consensus.cc:374] T f99d56107b6a48db9611f1ce8282d575 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:59.756737 17565 raft_consensus.cc:729] T f99d56107b6a48db9611f1ce8282d575 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cda358d5d184a4383a9716334390022, State: Initialized, Role: FOLLOWER
I20260807 03:37:59.756852 17563 tablet_bootstrap.cc:492] T efcd19aa11284c08ab3b4e1854a17dd9 P 2224f937b1b347e2931b7383c84894f2: Bootstrap starting.
I20260807 03:37:59.757409 17565 consensus_queue.cc:260] T f99d56107b6a48db9611f1ce8282d575 P 3cda358d5d184a4383a9716334390022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.759546 17565 ts_tablet_manager.cc:1428] T f99d56107b6a48db9611f1ce8282d575 P 3cda358d5d184a4383a9716334390022: Time spent starting tablet: real 0.006s user 0.002s sys 0.002s
I20260807 03:37:59.760521 17565 tablet_bootstrap.cc:492] T 64fc83d22ad945ad890e67f9763fe413 P 3cda358d5d184a4383a9716334390022: Bootstrap starting.
W20260807 03:37:59.766408 17313 leader_election.cc:341] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:37:59.766781 17565 tablet_bootstrap.cc:654] T 64fc83d22ad945ad890e67f9763fe413 P 3cda358d5d184a4383a9716334390022: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:59.767369 17563 tablet_bootstrap.cc:654] T efcd19aa11284c08ab3b4e1854a17dd9 P 2224f937b1b347e2931b7383c84894f2: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:59.780444 17574 leader_election.cc:288] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022 [CANDIDATE]: Term 1 election: Requested vote from peers 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887), 2224f937b1b347e2931b7383c84894f2 (127.15.198.131:41195)
I20260807 03:37:59.780683 17494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a41a7b13d3284dcd8b9b616eeb93c578" candidate_uuid: "3cda358d5d184a4383a9716334390022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2224f937b1b347e2931b7383c84894f2"
I20260807 03:37:59.781286 17494 raft_consensus.cc:3037] T a41a7b13d3284dcd8b9b616eeb93c578 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:37:59.783950 17565 tablet_bootstrap.cc:492] T 64fc83d22ad945ad890e67f9763fe413 P 3cda358d5d184a4383a9716334390022: No bootstrap required, opened a new log
I20260807 03:37:59.784435 17565 ts_tablet_manager.cc:1397] T 64fc83d22ad945ad890e67f9763fe413 P 3cda358d5d184a4383a9716334390022: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.009s
I20260807 03:37:59.785169 17563 tablet_bootstrap.cc:492] T efcd19aa11284c08ab3b4e1854a17dd9 P 2224f937b1b347e2931b7383c84894f2: No bootstrap required, opened a new log
I20260807 03:37:59.785619 17563 ts_tablet_manager.cc:1397] T efcd19aa11284c08ab3b4e1854a17dd9 P 2224f937b1b347e2931b7383c84894f2: Time spent bootstrapping tablet: real 0.029s user 0.016s sys 0.008s
I20260807 03:37:59.786049 17564 tablet_bootstrap.cc:492] T a41a7b13d3284dcd8b9b616eeb93c578 P 6c5dca27966a4b8cadb7cc0a8e0d1676: No bootstrap required, opened a new log
I20260807 03:37:59.786473 17564 ts_tablet_manager.cc:1397] T a41a7b13d3284dcd8b9b616eeb93c578 P 6c5dca27966a4b8cadb7cc0a8e0d1676: Time spent bootstrapping tablet: real 0.063s user 0.022s sys 0.022s
I20260807 03:37:59.786851 17565 raft_consensus.cc:348] T 64fc83d22ad945ad890e67f9763fe413 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.787531 17565 raft_consensus.cc:374] T 64fc83d22ad945ad890e67f9763fe413 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:59.787807 17565 raft_consensus.cc:729] T 64fc83d22ad945ad890e67f9763fe413 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cda358d5d184a4383a9716334390022, State: Initialized, Role: FOLLOWER
I20260807 03:37:59.788029 17494 raft_consensus.cc:2454] T a41a7b13d3284dcd8b9b616eeb93c578 P 2224f937b1b347e2931b7383c84894f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cda358d5d184a4383a9716334390022 in term 1.
I20260807 03:37:59.788460 17565 consensus_queue.cc:260] T 64fc83d22ad945ad890e67f9763fe413 P 3cda358d5d184a4383a9716334390022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.788614 17563 raft_consensus.cc:348] T efcd19aa11284c08ab3b4e1854a17dd9 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.789117 17316 leader_election.cc:302] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2224f937b1b347e2931b7383c84894f2, 3cda358d5d184a4383a9716334390022; no voters: 6c5dca27966a4b8cadb7cc0a8e0d1676
I20260807 03:37:59.789518 17563 raft_consensus.cc:374] T efcd19aa11284c08ab3b4e1854a17dd9 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:59.789927 17563 raft_consensus.cc:729] T efcd19aa11284c08ab3b4e1854a17dd9 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2224f937b1b347e2931b7383c84894f2, State: Initialized, Role: FOLLOWER
I20260807 03:37:59.790227 17574 raft_consensus.cc:2781] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:37:59.790647 17574 raft_consensus.cc:686] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022 [term 1 LEADER]: Becoming Leader. State: Replica: 3cda358d5d184a4383a9716334390022, State: Running, Role: LEADER
I20260807 03:37:59.790762 17565 ts_tablet_manager.cc:1428] T 64fc83d22ad945ad890e67f9763fe413 P 3cda358d5d184a4383a9716334390022: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260807 03:37:59.790673 17563 consensus_queue.cc:260] T efcd19aa11284c08ab3b4e1854a17dd9 P 2224f937b1b347e2931b7383c84894f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.791666 17565 tablet_bootstrap.cc:492] T efcd19aa11284c08ab3b4e1854a17dd9 P 3cda358d5d184a4383a9716334390022: Bootstrap starting.
I20260807 03:37:59.791484 17574 consensus_queue.cc:237] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.795905 17563 ts_tablet_manager.cc:1428] T efcd19aa11284c08ab3b4e1854a17dd9 P 2224f937b1b347e2931b7383c84894f2: Time spent starting tablet: real 0.010s user 0.004s sys 0.001s
I20260807 03:37:59.789031 17564 raft_consensus.cc:348] T a41a7b13d3284dcd8b9b616eeb93c578 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.796401 17564 raft_consensus.cc:374] T a41a7b13d3284dcd8b9b616eeb93c578 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:59.796706 17564 raft_consensus.cc:729] T a41a7b13d3284dcd8b9b616eeb93c578 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c5dca27966a4b8cadb7cc0a8e0d1676, State: Initialized, Role: FOLLOWER
I20260807 03:37:59.797084 17563 tablet_bootstrap.cc:492] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2: Bootstrap starting.
I20260807 03:37:59.797334 17564 consensus_queue.cc:260] T a41a7b13d3284dcd8b9b616eeb93c578 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.799476 17564 ts_tablet_manager.cc:1428] T a41a7b13d3284dcd8b9b616eeb93c578 P 6c5dca27966a4b8cadb7cc0a8e0d1676: Time spent starting tablet: real 0.013s user 0.004s sys 0.000s
I20260807 03:37:59.800560 17564 tablet_bootstrap.cc:492] T 80b452de6935470789989a31d5305c47 P 6c5dca27966a4b8cadb7cc0a8e0d1676: Bootstrap starting.
I20260807 03:37:59.803735 17565 tablet_bootstrap.cc:654] T efcd19aa11284c08ab3b4e1854a17dd9 P 3cda358d5d184a4383a9716334390022: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:59.806085 17564 tablet_bootstrap.cc:654] T 80b452de6935470789989a31d5305c47 P 6c5dca27966a4b8cadb7cc0a8e0d1676: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:59.809801 17080 catalog_manager.cc:5495] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cda358d5d184a4383a9716334390022 (127.15.198.130). New cstate: current_term: 1 leader_uuid: "3cda358d5d184a4383a9716334390022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } health_report { overall_health: UNKNOWN } } }
I20260807 03:37:59.821419 17565 tablet_bootstrap.cc:492] T efcd19aa11284c08ab3b4e1854a17dd9 P 3cda358d5d184a4383a9716334390022: No bootstrap required, opened a new log
I20260807 03:37:59.821841 17565 ts_tablet_manager.cc:1397] T efcd19aa11284c08ab3b4e1854a17dd9 P 3cda358d5d184a4383a9716334390022: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.007s
I20260807 03:37:59.823997 17563 tablet_bootstrap.cc:654] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:59.824066 17565 raft_consensus.cc:348] T efcd19aa11284c08ab3b4e1854a17dd9 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.824921 17565 raft_consensus.cc:374] T efcd19aa11284c08ab3b4e1854a17dd9 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:59.825239 17565 raft_consensus.cc:729] T efcd19aa11284c08ab3b4e1854a17dd9 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cda358d5d184a4383a9716334390022, State: Initialized, Role: FOLLOWER
I20260807 03:37:59.825901 17565 consensus_queue.cc:260] T efcd19aa11284c08ab3b4e1854a17dd9 P 3cda358d5d184a4383a9716334390022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.827095 17564 tablet_bootstrap.cc:492] T 80b452de6935470789989a31d5305c47 P 6c5dca27966a4b8cadb7cc0a8e0d1676: No bootstrap required, opened a new log
I20260807 03:37:59.827651 17565 ts_tablet_manager.cc:1428] T efcd19aa11284c08ab3b4e1854a17dd9 P 3cda358d5d184a4383a9716334390022: Time spent starting tablet: real 0.006s user 0.000s sys 0.004s
I20260807 03:37:59.827661 17564 ts_tablet_manager.cc:1397] T 80b452de6935470789989a31d5305c47 P 6c5dca27966a4b8cadb7cc0a8e0d1676: Time spent bootstrapping tablet: real 0.027s user 0.013s sys 0.013s
I20260807 03:37:59.828503 17565 tablet_bootstrap.cc:492] T c93d1700ec3248b3a169ff40a4186010 P 3cda358d5d184a4383a9716334390022: Bootstrap starting.
I20260807 03:37:59.831285 17564 raft_consensus.cc:348] T 80b452de6935470789989a31d5305c47 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.832260 17564 raft_consensus.cc:374] T 80b452de6935470789989a31d5305c47 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:59.832677 17564 raft_consensus.cc:729] T 80b452de6935470789989a31d5305c47 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c5dca27966a4b8cadb7cc0a8e0d1676, State: Initialized, Role: FOLLOWER
I20260807 03:37:59.834424 17565 tablet_bootstrap.cc:654] T c93d1700ec3248b3a169ff40a4186010 P 3cda358d5d184a4383a9716334390022: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:59.836043 17563 tablet_bootstrap.cc:492] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2: No bootstrap required, opened a new log
I20260807 03:37:59.836509 17563 ts_tablet_manager.cc:1397] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2: Time spent bootstrapping tablet: real 0.040s user 0.011s sys 0.008s
I20260807 03:37:59.833513 17564 consensus_queue.cc:260] T 80b452de6935470789989a31d5305c47 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.838275 17564 ts_tablet_manager.cc:1428] T 80b452de6935470789989a31d5305c47 P 6c5dca27966a4b8cadb7cc0a8e0d1676: Time spent starting tablet: real 0.010s user 0.003s sys 0.003s
I20260807 03:37:59.839334 17564 tablet_bootstrap.cc:492] T 96451ce78a6f45479178c8d596430538 P 6c5dca27966a4b8cadb7cc0a8e0d1676: Bootstrap starting.
I20260807 03:37:59.845137 17563 raft_consensus.cc:348] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.845794 17563 raft_consensus.cc:374] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:59.846073 17563 raft_consensus.cc:729] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2224f937b1b347e2931b7383c84894f2, State: Initialized, Role: FOLLOWER
I20260807 03:37:59.846292 17565 tablet_bootstrap.cc:492] T c93d1700ec3248b3a169ff40a4186010 P 3cda358d5d184a4383a9716334390022: No bootstrap required, opened a new log
I20260807 03:37:59.846670 17565 ts_tablet_manager.cc:1397] T c93d1700ec3248b3a169ff40a4186010 P 3cda358d5d184a4383a9716334390022: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.007s
I20260807 03:37:59.847252 17563 consensus_queue.cc:260] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.848198 17564 tablet_bootstrap.cc:654] T 96451ce78a6f45479178c8d596430538 P 6c5dca27966a4b8cadb7cc0a8e0d1676: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:59.848855 17565 raft_consensus.cc:348] T c93d1700ec3248b3a169ff40a4186010 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.849514 17565 raft_consensus.cc:374] T c93d1700ec3248b3a169ff40a4186010 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:59.849831 17565 raft_consensus.cc:729] T c93d1700ec3248b3a169ff40a4186010 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cda358d5d184a4383a9716334390022, State: Initialized, Role: FOLLOWER
I20260807 03:37:59.850476 17565 consensus_queue.cc:260] T c93d1700ec3248b3a169ff40a4186010 P 3cda358d5d184a4383a9716334390022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.851680 17600 raft_consensus.cc:482] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:37:59.852157 17600 raft_consensus.cc:504] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.854034 17600 leader_election.cc:288] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887), 3cda358d5d184a4383a9716334390022 (127.15.198.130:45465)
I20260807 03:37:59.854581 17565 ts_tablet_manager.cc:1428] T c93d1700ec3248b3a169ff40a4186010 P 3cda358d5d184a4383a9716334390022: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20260807 03:37:59.856132 17366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b452de6935470789989a31d5305c47" candidate_uuid: "2224f937b1b347e2931b7383c84894f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cda358d5d184a4383a9716334390022" is_pre_election: true
I20260807 03:37:59.856734 17366 raft_consensus.cc:2454] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2224f937b1b347e2931b7383c84894f2 in term 0.
I20260807 03:37:59.858352 17246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b452de6935470789989a31d5305c47" candidate_uuid: "2224f937b1b347e2931b7383c84894f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" is_pre_election: true
I20260807 03:37:59.858883 17246 raft_consensus.cc:2454] T 80b452de6935470789989a31d5305c47 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2224f937b1b347e2931b7383c84894f2 in term 0.
I20260807 03:37:59.859308 17564 tablet_bootstrap.cc:492] T 96451ce78a6f45479178c8d596430538 P 6c5dca27966a4b8cadb7cc0a8e0d1676: No bootstrap required, opened a new log
I20260807 03:37:59.859761 17564 ts_tablet_manager.cc:1397] T 96451ce78a6f45479178c8d596430538 P 6c5dca27966a4b8cadb7cc0a8e0d1676: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.007s
I20260807 03:37:59.859876 17440 leader_election.cc:302] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2224f937b1b347e2931b7383c84894f2, 6c5dca27966a4b8cadb7cc0a8e0d1676; no voters:
I20260807 03:37:59.860939 17574 raft_consensus.cc:482] T 6f881ad9daec4be7b7ef3b0475c45474 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:37:59.861624 17563 ts_tablet_manager.cc:1428] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2: Time spent starting tablet: real 0.025s user 0.006s sys 0.000s
I20260807 03:37:59.861368 17574 raft_consensus.cc:504] T 6f881ad9daec4be7b7ef3b0475c45474 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.862078 17564 raft_consensus.cc:348] T 96451ce78a6f45479178c8d596430538 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } }
I20260807 03:37:59.862730 17564 raft_consensus.cc:374] T 96451ce78a6f45479178c8d596430538 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:59.862999 17564 raft_consensus.cc:729] T 96451ce78a6f45479178c8d596430538 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c5dca27966a4b8cadb7cc0a8e0d1676, State: Initialized, Role: FOLLOWER
I20260807 03:37:59.863412 17574 leader_election.cc:288] T 6f881ad9daec4be7b7ef3b0475c45474 P 3cda358d5d184a4383a9716334390022 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887), 2224f937b1b347e2931b7383c84894f2 (127.15.198.131:41195)
I20260807 03:37:59.863852 17564 consensus_queue.cc:260] T 96451ce78a6f45479178c8d596430538 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } }
I20260807 03:37:59.865139 17563 tablet_bootstrap.cc:492] T 8f6f16d016d3437d8aeae4a64c42a856 P 2224f937b1b347e2931b7383c84894f2: Bootstrap starting.
I20260807 03:37:59.865418 17494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f881ad9daec4be7b7ef3b0475c45474" candidate_uuid: "3cda358d5d184a4383a9716334390022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2224f937b1b347e2931b7383c84894f2" is_pre_election: true
I20260807 03:37:59.866340 17564 ts_tablet_manager.cc:1428] T 96451ce78a6f45479178c8d596430538 P 6c5dca27966a4b8cadb7cc0a8e0d1676: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260807 03:37:59.866525 17316 leader_election.cc:341] T 6f881ad9daec4be7b7ef3b0475c45474 P 3cda358d5d184a4383a9716334390022 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2224f937b1b347e2931b7383c84894f2 (127.15.198.131:41195): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:37:59.867206 17564 tablet_bootstrap.cc:492] T c93d1700ec3248b3a169ff40a4186010 P 6c5dca27966a4b8cadb7cc0a8e0d1676: Bootstrap starting.
I20260807 03:37:59.866406 17246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f881ad9daec4be7b7ef3b0475c45474" candidate_uuid: "3cda358d5d184a4383a9716334390022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" is_pre_election: true
I20260807 03:37:59.869446 17246 raft_consensus.cc:2454] T 6f881ad9daec4be7b7ef3b0475c45474 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cda358d5d184a4383a9716334390022 in term 0.
I20260807 03:37:59.864853 17600 raft_consensus.cc:2781] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:37:59.870229 17600 raft_consensus.cc:482] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:37:59.870594 17563 tablet_bootstrap.cc:654] T 8f6f16d016d3437d8aeae4a64c42a856 P 2224f937b1b347e2931b7383c84894f2: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:59.870503 17313 leader_election.cc:302] T 6f881ad9daec4be7b7ef3b0475c45474 P 3cda358d5d184a4383a9716334390022 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3cda358d5d184a4383a9716334390022, 6c5dca27966a4b8cadb7cc0a8e0d1676; no voters: 2224f937b1b347e2931b7383c84894f2
I20260807 03:37:59.871861 17574 raft_consensus.cc:2781] T 6f881ad9daec4be7b7ef3b0475c45474 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:37:59.872243 17574 raft_consensus.cc:482] T 6f881ad9daec4be7b7ef3b0475c45474 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:37:59.872546 17574 raft_consensus.cc:3037] T 6f881ad9daec4be7b7ef3b0475c45474 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:37:59.872649 17564 tablet_bootstrap.cc:654] T c93d1700ec3248b3a169ff40a4186010 P 6c5dca27966a4b8cadb7cc0a8e0d1676: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:59.870543 17600 raft_consensus.cc:3037] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:37:59.880427 17600 raft_consensus.cc:504] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.881599 17574 raft_consensus.cc:504] T 6f881ad9daec4be7b7ef3b0475c45474 P 3cda358d5d184a4383a9716334390022 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.883143 17600 leader_election.cc:288] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 [CANDIDATE]: Term 1 election: Requested vote from peers 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887), 3cda358d5d184a4383a9716334390022 (127.15.198.130:45465)
I20260807 03:37:59.884243 17366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b452de6935470789989a31d5305c47" candidate_uuid: "2224f937b1b347e2931b7383c84894f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cda358d5d184a4383a9716334390022"
I20260807 03:37:59.884824 17366 raft_consensus.cc:3037] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:37:59.884557 17574 leader_election.cc:288] T 6f881ad9daec4be7b7ef3b0475c45474 P 3cda358d5d184a4383a9716334390022 [CANDIDATE]: Term 1 election: Requested vote from peers 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887), 2224f937b1b347e2931b7383c84894f2 (127.15.198.131:41195)
I20260807 03:37:59.884588 17246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f881ad9daec4be7b7ef3b0475c45474" candidate_uuid: "3cda358d5d184a4383a9716334390022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
I20260807 03:37:59.885397 17246 raft_consensus.cc:3037] T 6f881ad9daec4be7b7ef3b0475c45474 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:37:59.888266 17600 raft_consensus.cc:482] T c93d1700ec3248b3a169ff40a4186010 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:37:59.888710 17600 raft_consensus.cc:504] T c93d1700ec3248b3a169ff40a4186010 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.889060 17494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f881ad9daec4be7b7ef3b0475c45474" candidate_uuid: "3cda358d5d184a4383a9716334390022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2224f937b1b347e2931b7383c84894f2"
W20260807 03:37:59.890336 17316 leader_election.cc:341] T 6f881ad9daec4be7b7ef3b0475c45474 P 3cda358d5d184a4383a9716334390022 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2224f937b1b347e2931b7383c84894f2 (127.15.198.131:41195): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:37:59.890503 17600 leader_election.cc:288] T c93d1700ec3248b3a169ff40a4186010 P 2224f937b1b347e2931b7383c84894f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887), 3cda358d5d184a4383a9716334390022 (127.15.198.130:45465)
I20260807 03:37:59.891747 17367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c93d1700ec3248b3a169ff40a4186010" candidate_uuid: "2224f937b1b347e2931b7383c84894f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cda358d5d184a4383a9716334390022" is_pre_election: true
I20260807 03:37:59.892261 17247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b452de6935470789989a31d5305c47" candidate_uuid: "2224f937b1b347e2931b7383c84894f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
I20260807 03:37:59.892403 17367 raft_consensus.cc:2454] T c93d1700ec3248b3a169ff40a4186010 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2224f937b1b347e2931b7383c84894f2 in term 0.
I20260807 03:37:59.892786 17247 raft_consensus.cc:3037] T 80b452de6935470789989a31d5305c47 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:37:59.892519 17243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c93d1700ec3248b3a169ff40a4186010" candidate_uuid: "2224f937b1b347e2931b7383c84894f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" is_pre_election: true
I20260807 03:37:59.893543 17443 leader_election.cc:302] T c93d1700ec3248b3a169ff40a4186010 P 2224f937b1b347e2931b7383c84894f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2224f937b1b347e2931b7383c84894f2, 3cda358d5d184a4383a9716334390022; no voters:
W20260807 03:37:59.894145 17440 leader_election.cc:341] T c93d1700ec3248b3a169ff40a4186010 P 2224f937b1b347e2931b7383c84894f2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:37:59.894407 17600 raft_consensus.cc:2781] T c93d1700ec3248b3a169ff40a4186010 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:37:59.894865 17600 raft_consensus.cc:482] T c93d1700ec3248b3a169ff40a4186010 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:37:59.895236 17600 raft_consensus.cc:3037] T c93d1700ec3248b3a169ff40a4186010 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:37:59.900622 17247 raft_consensus.cc:2454] T 80b452de6935470789989a31d5305c47 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2224f937b1b347e2931b7383c84894f2 in term 1.
I20260807 03:37:59.901279 17600 raft_consensus.cc:504] T c93d1700ec3248b3a169ff40a4186010 P 2224f937b1b347e2931b7383c84894f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.901753 17366 raft_consensus.cc:2454] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2224f937b1b347e2931b7383c84894f2 in term 1.
I20260807 03:37:59.901916 17440 leader_election.cc:302] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2224f937b1b347e2931b7383c84894f2, 6c5dca27966a4b8cadb7cc0a8e0d1676; no voters:
I20260807 03:37:59.903489 17611 raft_consensus.cc:2781] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:37:59.904348 17600 leader_election.cc:288] T c93d1700ec3248b3a169ff40a4186010 P 2224f937b1b347e2931b7383c84894f2 [CANDIDATE]: Term 1 election: Requested vote from peers 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887), 3cda358d5d184a4383a9716334390022 (127.15.198.130:45465)
I20260807 03:37:59.903934 17611 raft_consensus.cc:686] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 [term 1 LEADER]: Becoming Leader. State: Replica: 2224f937b1b347e2931b7383c84894f2, State: Running, Role: LEADER
I20260807 03:37:59.905179 17564 tablet_bootstrap.cc:492] T c93d1700ec3248b3a169ff40a4186010 P 6c5dca27966a4b8cadb7cc0a8e0d1676: No bootstrap required, opened a new log
I20260807 03:37:59.905061 17247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c93d1700ec3248b3a169ff40a4186010" candidate_uuid: "2224f937b1b347e2931b7383c84894f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
I20260807 03:37:59.905609 17564 ts_tablet_manager.cc:1397] T c93d1700ec3248b3a169ff40a4186010 P 6c5dca27966a4b8cadb7cc0a8e0d1676: Time spent bootstrapping tablet: real 0.039s user 0.020s sys 0.008s
I20260807 03:37:59.905642 17366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c93d1700ec3248b3a169ff40a4186010" candidate_uuid: "2224f937b1b347e2931b7383c84894f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cda358d5d184a4383a9716334390022"
I20260807 03:37:59.906049 17607 raft_consensus.cc:482] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:37:59.906302 17366 raft_consensus.cc:3037] T c93d1700ec3248b3a169ff40a4186010 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:37:59.905786 17611 consensus_queue.cc:237] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.906587 17607 raft_consensus.cc:504] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.908011 17564 raft_consensus.cc:348] T c93d1700ec3248b3a169ff40a4186010 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.908594 17607 leader_election.cc:288] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cda358d5d184a4383a9716334390022 (127.15.198.130:45465), 2224f937b1b347e2931b7383c84894f2 (127.15.198.131:41195)
I20260807 03:37:59.908706 17564 raft_consensus.cc:374] T c93d1700ec3248b3a169ff40a4186010 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:59.908984 17564 raft_consensus.cc:729] T c93d1700ec3248b3a169ff40a4186010 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c5dca27966a4b8cadb7cc0a8e0d1676, State: Initialized, Role: FOLLOWER
I20260807 03:37:59.909557 17564 consensus_queue.cc:260] T c93d1700ec3248b3a169ff40a4186010 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.910241 17367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcd19aa11284c08ab3b4e1854a17dd9" candidate_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cda358d5d184a4383a9716334390022" is_pre_election: true
I20260807 03:37:59.910780 17367 raft_consensus.cc:2454] T efcd19aa11284c08ab3b4e1854a17dd9 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c5dca27966a4b8cadb7cc0a8e0d1676 in term 0.
I20260807 03:37:59.911458 17564 ts_tablet_manager.cc:1428] T c93d1700ec3248b3a169ff40a4186010 P 6c5dca27966a4b8cadb7cc0a8e0d1676: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s
I20260807 03:37:59.911816 17184 leader_election.cc:302] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cda358d5d184a4383a9716334390022, 6c5dca27966a4b8cadb7cc0a8e0d1676; no voters:
I20260807 03:37:59.912606 17607 raft_consensus.cc:2781] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:37:59.912976 17607 raft_consensus.cc:482] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:37:59.913360 17607 raft_consensus.cc:3037] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Advancing to term 1
W20260807 03:37:59.913995 17440 leader_election.cc:341] T c93d1700ec3248b3a169ff40a4186010 P 2224f937b1b347e2931b7383c84894f2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:37:59.915642 17494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcd19aa11284c08ab3b4e1854a17dd9" candidate_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2224f937b1b347e2931b7383c84894f2" is_pre_election: true
I20260807 03:37:59.916270 17494 raft_consensus.cc:2454] T efcd19aa11284c08ab3b4e1854a17dd9 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c5dca27966a4b8cadb7cc0a8e0d1676 in term 0.
I20260807 03:37:59.917862 17366 raft_consensus.cc:2454] T c93d1700ec3248b3a169ff40a4186010 P 3cda358d5d184a4383a9716334390022 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2224f937b1b347e2931b7383c84894f2 in term 1.
I20260807 03:37:59.918785 17246 raft_consensus.cc:2454] T 6f881ad9daec4be7b7ef3b0475c45474 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cda358d5d184a4383a9716334390022 in term 1.
I20260807 03:37:59.918959 17443 leader_election.cc:302] T c93d1700ec3248b3a169ff40a4186010 P 2224f937b1b347e2931b7383c84894f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2224f937b1b347e2931b7383c84894f2, 3cda358d5d184a4383a9716334390022; no voters: 6c5dca27966a4b8cadb7cc0a8e0d1676
I20260807 03:37:59.920924 17611 raft_consensus.cc:2781] T c93d1700ec3248b3a169ff40a4186010 P 2224f937b1b347e2931b7383c84894f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:37:59.921382 17611 raft_consensus.cc:686] T c93d1700ec3248b3a169ff40a4186010 P 2224f937b1b347e2931b7383c84894f2 [term 1 LEADER]: Becoming Leader. State: Replica: 2224f937b1b347e2931b7383c84894f2, State: Running, Role: LEADER
I20260807 03:37:59.919994 17563 tablet_bootstrap.cc:492] T 8f6f16d016d3437d8aeae4a64c42a856 P 2224f937b1b347e2931b7383c84894f2: No bootstrap required, opened a new log
I20260807 03:37:59.922616 17563 ts_tablet_manager.cc:1397] T 8f6f16d016d3437d8aeae4a64c42a856 P 2224f937b1b347e2931b7383c84894f2: Time spent bootstrapping tablet: real 0.058s user 0.011s sys 0.001s
I20260807 03:37:59.922231 17611 consensus_queue.cc:237] T c93d1700ec3248b3a169ff40a4186010 P 2224f937b1b347e2931b7383c84894f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.922230 17082 catalog_manager.cc:5495] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2224f937b1b347e2931b7383c84894f2 (127.15.198.131). New cstate: current_term: 1 leader_uuid: "2224f937b1b347e2931b7383c84894f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } health_report { overall_health: HEALTHY } } }
I20260807 03:37:59.924942 17563 raft_consensus.cc:348] T 8f6f16d016d3437d8aeae4a64c42a856 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.925575 17563 raft_consensus.cc:374] T 8f6f16d016d3437d8aeae4a64c42a856 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:59.925832 17563 raft_consensus.cc:729] T 8f6f16d016d3437d8aeae4a64c42a856 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2224f937b1b347e2931b7383c84894f2, State: Initialized, Role: FOLLOWER
I20260807 03:37:59.926416 17563 consensus_queue.cc:260] T 8f6f16d016d3437d8aeae4a64c42a856 P 2224f937b1b347e2931b7383c84894f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.926710 17313 leader_election.cc:302] T 6f881ad9daec4be7b7ef3b0475c45474 P 3cda358d5d184a4383a9716334390022 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3cda358d5d184a4383a9716334390022, 6c5dca27966a4b8cadb7cc0a8e0d1676; no voters: 2224f937b1b347e2931b7383c84894f2
I20260807 03:37:59.928210 17574 raft_consensus.cc:2781] T 6f881ad9daec4be7b7ef3b0475c45474 P 3cda358d5d184a4383a9716334390022 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:37:59.928508 17563 ts_tablet_manager.cc:1428] T 8f6f16d016d3437d8aeae4a64c42a856 P 2224f937b1b347e2931b7383c84894f2: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260807 03:37:59.928818 17574 raft_consensus.cc:686] T 6f881ad9daec4be7b7ef3b0475c45474 P 3cda358d5d184a4383a9716334390022 [term 1 LEADER]: Becoming Leader. State: Replica: 3cda358d5d184a4383a9716334390022, State: Running, Role: LEADER
I20260807 03:37:59.929337 17563 tablet_bootstrap.cc:492] T 6f881ad9daec4be7b7ef3b0475c45474 P 2224f937b1b347e2931b7383c84894f2: Bootstrap starting.
I20260807 03:37:59.929814 17574 consensus_queue.cc:237] T 6f881ad9daec4be7b7ef3b0475c45474 P 3cda358d5d184a4383a9716334390022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.934904 17563 tablet_bootstrap.cc:654] T 6f881ad9daec4be7b7ef3b0475c45474 P 2224f937b1b347e2931b7383c84894f2: Neither blocks nor log segments found. Creating new log.
I20260807 03:37:59.937870 17577 consensus_queue.cc:1035] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:37:59.939340 17607 raft_consensus.cc:504] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.941247 17607 leader_election.cc:288] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [CANDIDATE]: Term 1 election: Requested vote from peers 3cda358d5d184a4383a9716334390022 (127.15.198.130:45465), 2224f937b1b347e2931b7383c84894f2 (127.15.198.131:41195)
I20260807 03:37:59.942708 17496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcd19aa11284c08ab3b4e1854a17dd9" candidate_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2224f937b1b347e2931b7383c84894f2"
I20260807 03:37:59.943480 17496 raft_consensus.cc:3037] T efcd19aa11284c08ab3b4e1854a17dd9 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:37:59.946549 17366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcd19aa11284c08ab3b4e1854a17dd9" candidate_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cda358d5d184a4383a9716334390022"
I20260807 03:37:59.947319 17366 raft_consensus.cc:3037] T efcd19aa11284c08ab3b4e1854a17dd9 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:37:59.953547 17080 catalog_manager.cc:5495] T 6f881ad9daec4be7b7ef3b0475c45474 P 3cda358d5d184a4383a9716334390022 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cda358d5d184a4383a9716334390022 (127.15.198.130). New cstate: current_term: 1 leader_uuid: "3cda358d5d184a4383a9716334390022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } health_report { overall_health: UNKNOWN } } }
I20260807 03:37:59.953274 17081 catalog_manager.cc:5495] T c93d1700ec3248b3a169ff40a4186010 P 2224f937b1b347e2931b7383c84894f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2224f937b1b347e2931b7383c84894f2 (127.15.198.131). New cstate: current_term: 1 leader_uuid: "2224f937b1b347e2931b7383c84894f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } health_report { overall_health: HEALTHY } } }
I20260807 03:37:59.966478 17496 raft_consensus.cc:2454] T efcd19aa11284c08ab3b4e1854a17dd9 P 2224f937b1b347e2931b7383c84894f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c5dca27966a4b8cadb7cc0a8e0d1676 in term 1.
I20260807 03:37:59.967782 17184 leader_election.cc:302] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2224f937b1b347e2931b7383c84894f2, 6c5dca27966a4b8cadb7cc0a8e0d1676; no voters:
I20260807 03:37:59.969003 17607 raft_consensus.cc:2781] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:37:59.970659 17607 raft_consensus.cc:686] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 1 LEADER]: Becoming Leader. State: Replica: 6c5dca27966a4b8cadb7cc0a8e0d1676, State: Running, Role: LEADER
I20260807 03:37:59.971827 17607 consensus_queue.cc:237] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.974905 17563 tablet_bootstrap.cc:492] T 6f881ad9daec4be7b7ef3b0475c45474 P 2224f937b1b347e2931b7383c84894f2: No bootstrap required, opened a new log
I20260807 03:37:59.975328 17563 ts_tablet_manager.cc:1397] T 6f881ad9daec4be7b7ef3b0475c45474 P 2224f937b1b347e2931b7383c84894f2: Time spent bootstrapping tablet: real 0.046s user 0.030s sys 0.011s
I20260807 03:37:59.966997 17366 raft_consensus.cc:2454] T efcd19aa11284c08ab3b4e1854a17dd9 P 3cda358d5d184a4383a9716334390022 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c5dca27966a4b8cadb7cc0a8e0d1676 in term 1.
I20260807 03:37:59.977659 17563 raft_consensus.cc:348] T 6f881ad9daec4be7b7ef3b0475c45474 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.978322 17563 raft_consensus.cc:374] T 6f881ad9daec4be7b7ef3b0475c45474 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:37:59.978612 17563 raft_consensus.cc:729] T 6f881ad9daec4be7b7ef3b0475c45474 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2224f937b1b347e2931b7383c84894f2, State: Initialized, Role: FOLLOWER
I20260807 03:37:59.979279 17563 consensus_queue.cc:260] T 6f881ad9daec4be7b7ef3b0475c45474 P 2224f937b1b347e2931b7383c84894f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.981657 17563 ts_tablet_manager.cc:1428] T 6f881ad9daec4be7b7ef3b0475c45474 P 2224f937b1b347e2931b7383c84894f2: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260807 03:37:59.985713 17574 raft_consensus.cc:482] T f99d56107b6a48db9611f1ce8282d575 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:37:59.986320 17574 raft_consensus.cc:504] T f99d56107b6a48db9611f1ce8282d575 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.988747 17574 leader_election.cc:288] T f99d56107b6a48db9611f1ce8282d575 P 3cda358d5d184a4383a9716334390022 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887), 2224f937b1b347e2931b7383c84894f2 (127.15.198.131:41195)
I20260807 03:37:59.989481 17246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f99d56107b6a48db9611f1ce8282d575" candidate_uuid: "3cda358d5d184a4383a9716334390022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" is_pre_election: true
I20260807 03:37:59.990279 17246 raft_consensus.cc:2454] T f99d56107b6a48db9611f1ce8282d575 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cda358d5d184a4383a9716334390022 in term 0.
I20260807 03:37:59.990571 17611 raft_consensus.cc:482] T 8f6f16d016d3437d8aeae4a64c42a856 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:37:59.991034 17611 raft_consensus.cc:504] T 8f6f16d016d3437d8aeae4a64c42a856 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:37:59.991636 17313 leader_election.cc:302] T f99d56107b6a48db9611f1ce8282d575 P 3cda358d5d184a4383a9716334390022 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cda358d5d184a4383a9716334390022, 6c5dca27966a4b8cadb7cc0a8e0d1676; no voters:
I20260807 03:37:59.992654 17574 raft_consensus.cc:2781] T f99d56107b6a48db9611f1ce8282d575 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:37:59.993000 17611 leader_election.cc:288] T 8f6f16d016d3437d8aeae4a64c42a856 P 2224f937b1b347e2931b7383c84894f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887), 3cda358d5d184a4383a9716334390022 (127.15.198.130:45465)
I20260807 03:37:59.993053 17574 raft_consensus.cc:482] T f99d56107b6a48db9611f1ce8282d575 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:37:59.993587 17574 raft_consensus.cc:3037] T f99d56107b6a48db9611f1ce8282d575 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:37:59.993921 17246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f6f16d016d3437d8aeae4a64c42a856" candidate_uuid: "2224f937b1b347e2931b7383c84894f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" is_pre_election: true
I20260807 03:37:59.994576 17246 raft_consensus.cc:2454] T 8f6f16d016d3437d8aeae4a64c42a856 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2224f937b1b347e2931b7383c84894f2 in term 0.
I20260807 03:37:59.994488 17366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f6f16d016d3437d8aeae4a64c42a856" candidate_uuid: "2224f937b1b347e2931b7383c84894f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cda358d5d184a4383a9716334390022" is_pre_election: true
I20260807 03:37:59.995191 17366 raft_consensus.cc:2454] T 8f6f16d016d3437d8aeae4a64c42a856 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2224f937b1b347e2931b7383c84894f2 in term 0.
I20260807 03:37:59.995910 17440 leader_election.cc:302] T 8f6f16d016d3437d8aeae4a64c42a856 P 2224f937b1b347e2931b7383c84894f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2224f937b1b347e2931b7383c84894f2, 6c5dca27966a4b8cadb7cc0a8e0d1676; no voters:
I20260807 03:37:59.996951 17611 raft_consensus.cc:2781] T 8f6f16d016d3437d8aeae4a64c42a856 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:37:59.997299 17611 raft_consensus.cc:482] T 8f6f16d016d3437d8aeae4a64c42a856 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:37:59.997648 17611 raft_consensus.cc:3037] T 8f6f16d016d3437d8aeae4a64c42a856 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:00.000257 17574 raft_consensus.cc:504] T f99d56107b6a48db9611f1ce8282d575 P 3cda358d5d184a4383a9716334390022 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:00.002151 17574 leader_election.cc:288] T f99d56107b6a48db9611f1ce8282d575 P 3cda358d5d184a4383a9716334390022 [CANDIDATE]: Term 1 election: Requested vote from peers 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887), 2224f937b1b347e2931b7383c84894f2 (127.15.198.131:41195)
I20260807 03:38:00.002626 17574 raft_consensus.cc:482] T 64fc83d22ad945ad890e67f9763fe413 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:00.003317 17246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f99d56107b6a48db9611f1ce8282d575" candidate_uuid: "3cda358d5d184a4383a9716334390022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
I20260807 03:38:00.003427 17574 raft_consensus.cc:504] T 64fc83d22ad945ad890e67f9763fe413 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:00.004359 17246 raft_consensus.cc:3037] T f99d56107b6a48db9611f1ce8282d575 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:00.004971 17611 raft_consensus.cc:504] T 8f6f16d016d3437d8aeae4a64c42a856 P 2224f937b1b347e2931b7383c84894f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:00.005492 17496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f99d56107b6a48db9611f1ce8282d575" candidate_uuid: "3cda358d5d184a4383a9716334390022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2224f937b1b347e2931b7383c84894f2" is_pre_election: true
I20260807 03:38:00.005815 17574 leader_election.cc:288] T 64fc83d22ad945ad890e67f9763fe413 P 3cda358d5d184a4383a9716334390022 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887), 2224f937b1b347e2931b7383c84894f2 (127.15.198.131:41195)
I20260807 03:38:00.006119 17496 raft_consensus.cc:2454] T f99d56107b6a48db9611f1ce8282d575 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cda358d5d184a4383a9716334390022 in term 0.
I20260807 03:38:00.007076 17611 leader_election.cc:288] T 8f6f16d016d3437d8aeae4a64c42a856 P 2224f937b1b347e2931b7383c84894f2 [CANDIDATE]: Term 1 election: Requested vote from peers 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887), 3cda358d5d184a4383a9716334390022 (127.15.198.130:45465)
I20260807 03:38:00.007987 17243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f6f16d016d3437d8aeae4a64c42a856" candidate_uuid: "2224f937b1b347e2931b7383c84894f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
I20260807 03:38:00.008215 17247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64fc83d22ad945ad890e67f9763fe413" candidate_uuid: "3cda358d5d184a4383a9716334390022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" is_pre_election: true
I20260807 03:38:00.008639 17243 raft_consensus.cc:3037] T 8f6f16d016d3437d8aeae4a64c42a856 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:00.008842 17247 raft_consensus.cc:2454] T 64fc83d22ad945ad890e67f9763fe413 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cda358d5d184a4383a9716334390022 in term 0.
I20260807 03:38:00.010053 17313 leader_election.cc:302] T 64fc83d22ad945ad890e67f9763fe413 P 3cda358d5d184a4383a9716334390022 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cda358d5d184a4383a9716334390022, 6c5dca27966a4b8cadb7cc0a8e0d1676; no voters:
I20260807 03:38:00.011138 17246 raft_consensus.cc:2454] T f99d56107b6a48db9611f1ce8282d575 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cda358d5d184a4383a9716334390022 in term 1.
I20260807 03:38:00.011224 17493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64fc83d22ad945ad890e67f9763fe413" candidate_uuid: "3cda358d5d184a4383a9716334390022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2224f937b1b347e2931b7383c84894f2" is_pre_election: true
I20260807 03:38:00.011932 17493 raft_consensus.cc:2454] T 64fc83d22ad945ad890e67f9763fe413 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cda358d5d184a4383a9716334390022 in term 0.
I20260807 03:38:00.012338 17313 leader_election.cc:302] T f99d56107b6a48db9611f1ce8282d575 P 3cda358d5d184a4383a9716334390022 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cda358d5d184a4383a9716334390022, 6c5dca27966a4b8cadb7cc0a8e0d1676; no voters:
I20260807 03:38:00.013242 17496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f99d56107b6a48db9611f1ce8282d575" candidate_uuid: "3cda358d5d184a4383a9716334390022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2224f937b1b347e2931b7383c84894f2"
I20260807 03:38:00.013969 17496 raft_consensus.cc:3037] T f99d56107b6a48db9611f1ce8282d575 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:00.016274 17573 raft_consensus.cc:482] T c93d1700ec3248b3a169ff40a4186010 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:00.016732 17573 raft_consensus.cc:504] T c93d1700ec3248b3a169ff40a4186010 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:00.017340 17243 raft_consensus.cc:2454] T 8f6f16d016d3437d8aeae4a64c42a856 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2224f937b1b347e2931b7383c84894f2 in term 1.
I20260807 03:38:00.018697 17573 leader_election.cc:288] T c93d1700ec3248b3a169ff40a4186010 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cda358d5d184a4383a9716334390022 (127.15.198.130:45465), 2224f937b1b347e2931b7383c84894f2 (127.15.198.131:41195)
I20260807 03:38:00.018657 17440 leader_election.cc:302] T 8f6f16d016d3437d8aeae4a64c42a856 P 2224f937b1b347e2931b7383c84894f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2224f937b1b347e2931b7383c84894f2, 6c5dca27966a4b8cadb7cc0a8e0d1676; no voters:
I20260807 03:38:00.020692 17654 raft_consensus.cc:2781] T 64fc83d22ad945ad890e67f9763fe413 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:00.022310 17654 raft_consensus.cc:482] T 64fc83d22ad945ad890e67f9763fe413 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:00.023046 17654 raft_consensus.cc:3037] T 64fc83d22ad945ad890e67f9763fe413 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:00.021272 17493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c93d1700ec3248b3a169ff40a4186010" candidate_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2224f937b1b347e2931b7383c84894f2" is_pre_election: true
I20260807 03:38:00.021835 17611 raft_consensus.cc:2781] T 8f6f16d016d3437d8aeae4a64c42a856 P 2224f937b1b347e2931b7383c84894f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:00.027168 17611 raft_consensus.cc:686] T 8f6f16d016d3437d8aeae4a64c42a856 P 2224f937b1b347e2931b7383c84894f2 [term 1 LEADER]: Becoming Leader. State: Replica: 2224f937b1b347e2931b7383c84894f2, State: Running, Role: LEADER
I20260807 03:38:00.021733 17366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f6f16d016d3437d8aeae4a64c42a856" candidate_uuid: "2224f937b1b347e2931b7383c84894f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cda358d5d184a4383a9716334390022"
I20260807 03:38:00.023141 17367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c93d1700ec3248b3a169ff40a4186010" candidate_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cda358d5d184a4383a9716334390022" is_pre_election: true
I20260807 03:38:00.028343 17367 raft_consensus.cc:2379] T c93d1700ec3248b3a169ff40a4186010 P 3cda358d5d184a4383a9716334390022 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6c5dca27966a4b8cadb7cc0a8e0d1676 in current term 1: Already voted for candidate 2224f937b1b347e2931b7383c84894f2 in this term.
I20260807 03:38:00.028057 17611 consensus_queue.cc:237] T 8f6f16d016d3437d8aeae4a64c42a856 P 2224f937b1b347e2931b7383c84894f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:00.029049 17366 raft_consensus.cc:3037] T 8f6f16d016d3437d8aeae4a64c42a856 P 3cda358d5d184a4383a9716334390022 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:00.024617 17574 raft_consensus.cc:2781] T f99d56107b6a48db9611f1ce8282d575 P 3cda358d5d184a4383a9716334390022 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:00.036584 17184 leader_election.cc:302] T c93d1700ec3248b3a169ff40a4186010 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c5dca27966a4b8cadb7cc0a8e0d1676; no voters: 2224f937b1b347e2931b7383c84894f2, 3cda358d5d184a4383a9716334390022
I20260807 03:38:00.036639 17080 catalog_manager.cc:5495] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129). New cstate: current_term: 1 leader_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:00.037979 17573 raft_consensus.cc:3037] T c93d1700ec3248b3a169ff40a4186010 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:00.037685 17654 raft_consensus.cc:504] T 64fc83d22ad945ad890e67f9763fe413 P 3cda358d5d184a4383a9716334390022 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:00.021067 17496 raft_consensus.cc:2454] T f99d56107b6a48db9611f1ce8282d575 P 2224f937b1b347e2931b7383c84894f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cda358d5d184a4383a9716334390022 in term 1.
I20260807 03:38:00.041213 17654 leader_election.cc:288] T 64fc83d22ad945ad890e67f9763fe413 P 3cda358d5d184a4383a9716334390022 [CANDIDATE]: Term 1 election: Requested vote from peers 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887), 2224f937b1b347e2931b7383c84894f2 (127.15.198.131:41195)
I20260807 03:38:00.045346 17246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64fc83d22ad945ad890e67f9763fe413" candidate_uuid: "3cda358d5d184a4383a9716334390022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
I20260807 03:38:00.046092 17246 raft_consensus.cc:3037] T 64fc83d22ad945ad890e67f9763fe413 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:00.052488 17574 raft_consensus.cc:686] T f99d56107b6a48db9611f1ce8282d575 P 3cda358d5d184a4383a9716334390022 [term 1 LEADER]: Becoming Leader. State: Replica: 3cda358d5d184a4383a9716334390022, State: Running, Role: LEADER
I20260807 03:38:00.055778 17366 raft_consensus.cc:2454] T 8f6f16d016d3437d8aeae4a64c42a856 P 3cda358d5d184a4383a9716334390022 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2224f937b1b347e2931b7383c84894f2 in term 1.
I20260807 03:38:00.055925 17573 raft_consensus.cc:2726] T c93d1700ec3248b3a169ff40a4186010 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:00.056578 17243 raft_consensus.cc:3037] T 96451ce78a6f45479178c8d596430538 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:00.053373 17574 consensus_queue.cc:237] T f99d56107b6a48db9611f1ce8282d575 P 3cda358d5d184a4383a9716334390022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:00.058715 17246 raft_consensus.cc:2454] T 64fc83d22ad945ad890e67f9763fe413 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cda358d5d184a4383a9716334390022 in term 1.
I20260807 03:38:00.061447 17496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64fc83d22ad945ad890e67f9763fe413" candidate_uuid: "3cda358d5d184a4383a9716334390022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2224f937b1b347e2931b7383c84894f2"
I20260807 03:38:00.062042 17496 raft_consensus.cc:3037] T 64fc83d22ad945ad890e67f9763fe413 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:00.066685 17080 catalog_manager.cc:5495] T 8f6f16d016d3437d8aeae4a64c42a856 P 2224f937b1b347e2931b7383c84894f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2224f937b1b347e2931b7383c84894f2 (127.15.198.131). New cstate: current_term: 1 leader_uuid: "2224f937b1b347e2931b7383c84894f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } health_report { overall_health: HEALTHY } } }
I20260807 03:38:00.067255 17654 consensus_queue.cc:1035] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 03:38:00.068293 17313 leader_election.cc:302] T 64fc83d22ad945ad890e67f9763fe413 P 3cda358d5d184a4383a9716334390022 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cda358d5d184a4383a9716334390022, 6c5dca27966a4b8cadb7cc0a8e0d1676; no voters:
I20260807 03:38:00.069087 17654 raft_consensus.cc:2781] T 64fc83d22ad945ad890e67f9763fe413 P 3cda358d5d184a4383a9716334390022 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:00.069545 17654 raft_consensus.cc:686] T 64fc83d22ad945ad890e67f9763fe413 P 3cda358d5d184a4383a9716334390022 [term 1 LEADER]: Becoming Leader. State: Replica: 3cda358d5d184a4383a9716334390022, State: Running, Role: LEADER
I20260807 03:38:00.070308 17654 consensus_queue.cc:237] T 64fc83d22ad945ad890e67f9763fe413 P 3cda358d5d184a4383a9716334390022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:00.074287 17496 raft_consensus.cc:2454] T 64fc83d22ad945ad890e67f9763fe413 P 2224f937b1b347e2931b7383c84894f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cda358d5d184a4383a9716334390022 in term 1.
I20260807 03:38:00.089674 17081 catalog_manager.cc:5495] T f99d56107b6a48db9611f1ce8282d575 P 3cda358d5d184a4383a9716334390022 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cda358d5d184a4383a9716334390022 (127.15.198.130). New cstate: current_term: 1 leader_uuid: "3cda358d5d184a4383a9716334390022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:00.102626 17081 catalog_manager.cc:5495] T 64fc83d22ad945ad890e67f9763fe413 P 3cda358d5d184a4383a9716334390022 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cda358d5d184a4383a9716334390022 (127.15.198.130). New cstate: current_term: 1 leader_uuid: "3cda358d5d184a4383a9716334390022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:00.190330 17243 raft_consensus.cc:1226] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 FOLLOWER]: Rejecting Update request from peer 2224f937b1b347e2931b7383c84894f2 for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:00.191522 17611 consensus_queue.cc:1046] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:00.192343 17600 raft_consensus.cc:3032] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:00.192595 17600 raft_consensus.cc:729] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2224f937b1b347e2931b7383c84894f2, State: Running, Role: LEADER
I20260807 03:38:00.193128 17600 consensus_queue.cc:260] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:00.193902 17600 raft_consensus.cc:3037] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:00.284013 17573 raft_consensus.cc:482] T a41a7b13d3284dcd8b9b616eeb93c578 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:00.284556 17573 raft_consensus.cc:504] T a41a7b13d3284dcd8b9b616eeb93c578 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:00.286077 17573 leader_election.cc:288] T a41a7b13d3284dcd8b9b616eeb93c578 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cda358d5d184a4383a9716334390022 (127.15.198.130:45465), 2224f937b1b347e2931b7383c84894f2 (127.15.198.131:41195)
I20260807 03:38:00.286991 17366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a41a7b13d3284dcd8b9b616eeb93c578" candidate_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cda358d5d184a4383a9716334390022" is_pre_election: true
I20260807 03:38:00.287127 17496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a41a7b13d3284dcd8b9b616eeb93c578" candidate_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2224f937b1b347e2931b7383c84894f2" is_pre_election: true
I20260807 03:38:00.287825 17496 raft_consensus.cc:2379] T a41a7b13d3284dcd8b9b616eeb93c578 P 2224f937b1b347e2931b7383c84894f2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6c5dca27966a4b8cadb7cc0a8e0d1676 in current term 1: Already voted for candidate 3cda358d5d184a4383a9716334390022 in this term.
I20260807 03:38:00.288671 17184 leader_election.cc:302] T a41a7b13d3284dcd8b9b616eeb93c578 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c5dca27966a4b8cadb7cc0a8e0d1676; no voters: 2224f937b1b347e2931b7383c84894f2, 3cda358d5d184a4383a9716334390022
I20260807 03:38:00.289264 17573 raft_consensus.cc:3037] T a41a7b13d3284dcd8b9b616eeb93c578 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:00.293874 17573 raft_consensus.cc:2726] T a41a7b13d3284dcd8b9b616eeb93c578 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:00.307089 17600 consensus_queue.cc:1035] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:00.311223 17611 consensus_queue.cc:1035] T c93d1700ec3248b3a169ff40a4186010 P 2224f937b1b347e2931b7383c84894f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:00.319563 17600 consensus_queue.cc:1035] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:00.334020 17600 consensus_queue.cc:1035] T c93d1700ec3248b3a169ff40a4186010 P 2224f937b1b347e2931b7383c84894f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:00.347806 17574 consensus_queue.cc:1035] T 6f881ad9daec4be7b7ef3b0475c45474 P 3cda358d5d184a4383a9716334390022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:00.357414 17496 raft_consensus.cc:3037] T 6f881ad9daec4be7b7ef3b0475c45474 P 2224f937b1b347e2931b7383c84894f2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:00.364220 17367 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "96451ce78a6f45479178c8d596430538"
dest_uuid: "3cda358d5d184a4383a9716334390022"
mode: GRACEFUL
new_leader_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
from {username='slave'} at 127.0.0.1:52436
I20260807 03:38:00.364959 17367 raft_consensus.cc:595] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022 [term 1 LEADER]: Received request to transfer leadership to TS 6c5dca27966a4b8cadb7cc0a8e0d1676
I20260807 03:38:00.366657 17657 consensus_queue.cc:1035] T 6f881ad9daec4be7b7ef3b0475c45474 P 3cda358d5d184a4383a9716334390022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:00.377027 17657 consensus_queue.cc:1035] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:00.389228 17657 consensus_queue.cc:1035] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:00.390242 17496 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e3f7b02979d648108ef56a2acf13fbf0"
dest_uuid: "2224f937b1b347e2931b7383c84894f2"
mode: GRACEFUL
new_leader_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
from {username='slave'} at 127.0.0.1:48322
I20260807 03:38:00.391168 17496 raft_consensus.cc:595] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 2 FOLLOWER]: Received request to transfer leadership to TS 6c5dca27966a4b8cadb7cc0a8e0d1676
I20260807 03:38:00.391515 17496 raft_consensus.cc:603] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 03:38:00.397205 17367 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f99d56107b6a48db9611f1ce8282d575"
dest_uuid: "3cda358d5d184a4383a9716334390022"
mode: GRACEFUL
new_leader_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
from {username='slave'} at 127.0.0.1:52436
I20260807 03:38:00.397652 17367 raft_consensus.cc:595] T f99d56107b6a48db9611f1ce8282d575 P 3cda358d5d184a4383a9716334390022 [term 1 LEADER]: Received request to transfer leadership to TS 6c5dca27966a4b8cadb7cc0a8e0d1676
I20260807 03:38:00.400981 17573 consensus_queue.cc:1035] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:00.403012 17493 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8f6f16d016d3437d8aeae4a64c42a856"
dest_uuid: "2224f937b1b347e2931b7383c84894f2"
mode: GRACEFUL
new_leader_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
from {username='slave'} at 127.0.0.1:48322
I20260807 03:38:00.403626 17493 raft_consensus.cc:595] T 8f6f16d016d3437d8aeae4a64c42a856 P 2224f937b1b347e2931b7383c84894f2 [term 1 LEADER]: Received request to transfer leadership to TS 6c5dca27966a4b8cadb7cc0a8e0d1676
I20260807 03:38:00.408614 17367 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a41a7b13d3284dcd8b9b616eeb93c578"
dest_uuid: "3cda358d5d184a4383a9716334390022"
mode: GRACEFUL
new_leader_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
from {username='slave'} at 127.0.0.1:52436
I20260807 03:38:00.409068 17367 raft_consensus.cc:595] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022 [term 1 LEADER]: Received request to transfer leadership to TS 6c5dca27966a4b8cadb7cc0a8e0d1676
I20260807 03:38:00.413738 17496 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "80b452de6935470789989a31d5305c47"
dest_uuid: "2224f937b1b347e2931b7383c84894f2"
mode: GRACEFUL
new_leader_uuid: "3cda358d5d184a4383a9716334390022"
from {username='slave'} at 127.0.0.1:48322
I20260807 03:38:00.414304 17496 raft_consensus.cc:595] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 [term 1 LEADER]: Received request to transfer leadership to TS 3cda358d5d184a4383a9716334390022
I20260807 03:38:00.418380 17607 consensus_queue.cc:1035] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:00.437891 17243 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "efcd19aa11284c08ab3b4e1854a17dd9"
dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
mode: GRACEFUL
new_leader_uuid: "3cda358d5d184a4383a9716334390022"
from {username='slave'} at 127.0.0.1:60374
I20260807 03:38:00.438421 17243 raft_consensus.cc:595] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 1 LEADER]: Received request to transfer leadership to TS 3cda358d5d184a4383a9716334390022
I20260807 03:38:00.494972 17657 consensus_queue.cc:1035] T f99d56107b6a48db9611f1ce8282d575 P 3cda358d5d184a4383a9716334390022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:00.501451 17656 raft_consensus.cc:979] T f99d56107b6a48db9611f1ce8282d575 P 3cda358d5d184a4383a9716334390022: : Instructing follower 6c5dca27966a4b8cadb7cc0a8e0d1676 to start an election
I20260807 03:38:00.501547 17572 consensus_queue.cc:1035] T 8f6f16d016d3437d8aeae4a64c42a856 P 2224f937b1b347e2931b7383c84894f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:00.501880 17656 raft_consensus.cc:1067] T f99d56107b6a48db9611f1ce8282d575 P 3cda358d5d184a4383a9716334390022 [term 1 LEADER]: Signalling peer 6c5dca27966a4b8cadb7cc0a8e0d1676 to start an election
I20260807 03:38:00.503753 17243 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f99d56107b6a48db9611f1ce8282d575"
dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
from {username='slave'} at 127.0.0.1:60348
I20260807 03:38:00.508955 17243 raft_consensus.cc:482] T f99d56107b6a48db9611f1ce8282d575 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:38:00.509327 17243 raft_consensus.cc:3037] T f99d56107b6a48db9611f1ce8282d575 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:00.509609 17673 raft_consensus.cc:979] T 8f6f16d016d3437d8aeae4a64c42a856 P 2224f937b1b347e2931b7383c84894f2: : Instructing follower 6c5dca27966a4b8cadb7cc0a8e0d1676 to start an election
I20260807 03:38:00.508198 17657 consensus_queue.cc:1035] T f99d56107b6a48db9611f1ce8282d575 P 3cda358d5d184a4383a9716334390022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:00.510615 17572 raft_consensus.cc:1067] T 8f6f16d016d3437d8aeae4a64c42a856 P 2224f937b1b347e2931b7383c84894f2 [term 1 LEADER]: Signalling peer 6c5dca27966a4b8cadb7cc0a8e0d1676 to start an election
I20260807 03:38:00.515951 17600 consensus_queue.cc:1035] T 8f6f16d016d3437d8aeae4a64c42a856 P 2224f937b1b347e2931b7383c84894f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:00.518448 17242 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8f6f16d016d3437d8aeae4a64c42a856"
dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
from {username='slave'} at 127.0.0.1:60360
I20260807 03:38:00.519106 17242 raft_consensus.cc:482] T 8f6f16d016d3437d8aeae4a64c42a856 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:38:00.519461 17242 raft_consensus.cc:3037] T 8f6f16d016d3437d8aeae4a64c42a856 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:00.518488 17243 raft_consensus.cc:504] T f99d56107b6a48db9611f1ce8282d575 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:00.522100 17243 leader_election.cc:288] T f99d56107b6a48db9611f1ce8282d575 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [CANDIDATE]: Term 2 election: Requested vote from peers 3cda358d5d184a4383a9716334390022 (127.15.198.130:45465), 2224f937b1b347e2931b7383c84894f2 (127.15.198.131:41195)
I20260807 03:38:00.523021 17493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f99d56107b6a48db9611f1ce8282d575" candidate_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2224f937b1b347e2931b7383c84894f2"
I20260807 03:38:00.523033 17366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f99d56107b6a48db9611f1ce8282d575" candidate_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3cda358d5d184a4383a9716334390022"
I20260807 03:38:00.523797 17493 raft_consensus.cc:3037] T f99d56107b6a48db9611f1ce8282d575 P 2224f937b1b347e2931b7383c84894f2 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:00.523962 17366 raft_consensus.cc:3032] T f99d56107b6a48db9611f1ce8282d575 P 3cda358d5d184a4383a9716334390022 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:00.524354 17366 raft_consensus.cc:729] T f99d56107b6a48db9611f1ce8282d575 P 3cda358d5d184a4383a9716334390022 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3cda358d5d184a4383a9716334390022, State: Running, Role: LEADER
I20260807 03:38:00.525053 17366 consensus_queue.cc:260] T f99d56107b6a48db9611f1ce8282d575 P 3cda358d5d184a4383a9716334390022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:00.526074 17366 raft_consensus.cc:3037] T f99d56107b6a48db9611f1ce8282d575 P 3cda358d5d184a4383a9716334390022 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:00.526211 17246 raft_consensus.cc:1226] T f99d56107b6a48db9611f1ce8282d575 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 FOLLOWER]: Rejecting Update request from peer 3cda358d5d184a4383a9716334390022 for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:00.527249 17242 raft_consensus.cc:504] T 8f6f16d016d3437d8aeae4a64c42a856 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:00.530517 17493 raft_consensus.cc:2454] T f99d56107b6a48db9611f1ce8282d575 P 2224f937b1b347e2931b7383c84894f2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c5dca27966a4b8cadb7cc0a8e0d1676 in term 2.
I20260807 03:38:00.531255 17364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f6f16d016d3437d8aeae4a64c42a856" candidate_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3cda358d5d184a4383a9716334390022"
I20260807 03:38:00.531708 17184 leader_election.cc:302] T f99d56107b6a48db9611f1ce8282d575 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2224f937b1b347e2931b7383c84894f2, 6c5dca27966a4b8cadb7cc0a8e0d1676; no voters:
I20260807 03:38:00.532506 17607 raft_consensus.cc:2781] T f99d56107b6a48db9611f1ce8282d575 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:38:00.532903 17607 raft_consensus.cc:686] T f99d56107b6a48db9611f1ce8282d575 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 LEADER]: Becoming Leader. State: Replica: 6c5dca27966a4b8cadb7cc0a8e0d1676, State: Running, Role: LEADER
I20260807 03:38:00.533635 17607 consensus_queue.cc:237] T f99d56107b6a48db9611f1ce8282d575 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:00.534878 17493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f6f16d016d3437d8aeae4a64c42a856" candidate_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2224f937b1b347e2931b7383c84894f2"
I20260807 03:38:00.536760 17493 raft_consensus.cc:3032] T 8f6f16d016d3437d8aeae4a64c42a856 P 2224f937b1b347e2931b7383c84894f2 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:00.537046 17493 raft_consensus.cc:729] T 8f6f16d016d3437d8aeae4a64c42a856 P 2224f937b1b347e2931b7383c84894f2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2224f937b1b347e2931b7383c84894f2, State: Running, Role: LEADER
I20260807 03:38:00.536371 17242 leader_election.cc:288] T 8f6f16d016d3437d8aeae4a64c42a856 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [CANDIDATE]: Term 2 election: Requested vote from peers 3cda358d5d184a4383a9716334390022 (127.15.198.130:45465), 2224f937b1b347e2931b7383c84894f2 (127.15.198.131:41195)
I20260807 03:38:00.539108 17366 raft_consensus.cc:2454] T f99d56107b6a48db9611f1ce8282d575 P 3cda358d5d184a4383a9716334390022 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c5dca27966a4b8cadb7cc0a8e0d1676 in term 2.
I20260807 03:38:00.540705 17493 consensus_queue.cc:260] T 8f6f16d016d3437d8aeae4a64c42a856 P 2224f937b1b347e2931b7383c84894f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:00.541615 17493 raft_consensus.cc:3037] T 8f6f16d016d3437d8aeae4a64c42a856 P 2224f937b1b347e2931b7383c84894f2 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:00.546465 17493 raft_consensus.cc:2454] T 8f6f16d016d3437d8aeae4a64c42a856 P 2224f937b1b347e2931b7383c84894f2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c5dca27966a4b8cadb7cc0a8e0d1676 in term 2.
I20260807 03:38:00.547549 17184 leader_election.cc:302] T 8f6f16d016d3437d8aeae4a64c42a856 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2224f937b1b347e2931b7383c84894f2, 6c5dca27966a4b8cadb7cc0a8e0d1676; no voters: 3cda358d5d184a4383a9716334390022
I20260807 03:38:00.548696 17607 raft_consensus.cc:2781] T 8f6f16d016d3437d8aeae4a64c42a856 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:38:00.548632 17082 catalog_manager.cc:5495] T f99d56107b6a48db9611f1ce8282d575 P 6c5dca27966a4b8cadb7cc0a8e0d1676 reported cstate change: term changed from 1 to 2, leader changed from 3cda358d5d184a4383a9716334390022 (127.15.198.130) to 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129). New cstate: current_term: 2 leader_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:00.549407 17607 raft_consensus.cc:686] T 8f6f16d016d3437d8aeae4a64c42a856 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 LEADER]: Becoming Leader. State: Replica: 6c5dca27966a4b8cadb7cc0a8e0d1676, State: Running, Role: LEADER
I20260807 03:38:00.550211 17656 raft_consensus.cc:979] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022: : Instructing follower 6c5dca27966a4b8cadb7cc0a8e0d1676 to start an election
I20260807 03:38:00.550719 17657 raft_consensus.cc:1067] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022 [term 1 LEADER]: Signalling peer 6c5dca27966a4b8cadb7cc0a8e0d1676 to start an election
I20260807 03:38:00.550314 17607 consensus_queue.cc:237] T 8f6f16d016d3437d8aeae4a64c42a856 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:00.552232 17242 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "96451ce78a6f45479178c8d596430538"
dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
from {username='slave'} at 127.0.0.1:60348
I20260807 03:38:00.552798 17242 raft_consensus.cc:482] T 96451ce78a6f45479178c8d596430538 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:38:00.553145 17242 raft_consensus.cc:3037] T 96451ce78a6f45479178c8d596430538 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:00.559988 17242 raft_consensus.cc:504] T 96451ce78a6f45479178c8d596430538 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } }
I20260807 03:38:00.562196 17242 leader_election.cc:288] T 96451ce78a6f45479178c8d596430538 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [CANDIDATE]: Term 2 election: Requested vote from peers 2224f937b1b347e2931b7383c84894f2 (127.15.198.131:41195), 3cda358d5d184a4383a9716334390022 (127.15.198.130:45465)
I20260807 03:38:00.562815 17493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96451ce78a6f45479178c8d596430538" candidate_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2224f937b1b347e2931b7383c84894f2"
I20260807 03:38:00.563428 17493 raft_consensus.cc:3037] T 96451ce78a6f45479178c8d596430538 P 2224f937b1b347e2931b7383c84894f2 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:00.563407 17367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96451ce78a6f45479178c8d596430538" candidate_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3cda358d5d184a4383a9716334390022"
I20260807 03:38:00.563187 17082 catalog_manager.cc:5495] T 8f6f16d016d3437d8aeae4a64c42a856 P 6c5dca27966a4b8cadb7cc0a8e0d1676 reported cstate change: term changed from 1 to 2, leader changed from 2224f937b1b347e2931b7383c84894f2 (127.15.198.131) to 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129). New cstate: current_term: 2 leader_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:00.564131 17367 raft_consensus.cc:3032] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:00.564499 17367 raft_consensus.cc:729] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3cda358d5d184a4383a9716334390022, State: Running, Role: LEADER
I20260807 03:38:00.565212 17367 consensus_queue.cc:260] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } }
I20260807 03:38:00.566419 17367 raft_consensus.cc:3037] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:00.570112 17493 raft_consensus.cc:2454] T 96451ce78a6f45479178c8d596430538 P 2224f937b1b347e2931b7383c84894f2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c5dca27966a4b8cadb7cc0a8e0d1676 in term 2.
I20260807 03:38:00.571208 17184 leader_election.cc:302] T 96451ce78a6f45479178c8d596430538 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2224f937b1b347e2931b7383c84894f2, 6c5dca27966a4b8cadb7cc0a8e0d1676; no voters:
I20260807 03:38:00.572011 17607 raft_consensus.cc:2781] T 96451ce78a6f45479178c8d596430538 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:38:00.572525 17607 raft_consensus.cc:686] T 96451ce78a6f45479178c8d596430538 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 LEADER]: Becoming Leader. State: Replica: 6c5dca27966a4b8cadb7cc0a8e0d1676, State: Running, Role: LEADER
I20260807 03:38:00.573422 17367 raft_consensus.cc:2454] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c5dca27966a4b8cadb7cc0a8e0d1676 in term 2.
I20260807 03:38:00.573417 17607 consensus_queue.cc:237] T 96451ce78a6f45479178c8d596430538 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } }
I20260807 03:38:00.579934 17082 catalog_manager.cc:5495] T 96451ce78a6f45479178c8d596430538 P 6c5dca27966a4b8cadb7cc0a8e0d1676 reported cstate change: term changed from 1 to 2, leader changed from 3cda358d5d184a4383a9716334390022 (127.15.198.130) to 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129). New cstate: current_term: 2 leader_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } health_report { overall_health: HEALTHY } } }
I20260807 03:38:00.610309 17657 consensus_queue.cc:1035] T 64fc83d22ad945ad890e67f9763fe413 P 3cda358d5d184a4383a9716334390022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:00.620527 17574 consensus_queue.cc:1035] T 64fc83d22ad945ad890e67f9763fe413 P 3cda358d5d184a4383a9716334390022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:00.710969 17611 raft_consensus.cc:979] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2: : Instructing follower 3cda358d5d184a4383a9716334390022 to start an election
I20260807 03:38:00.711442 17600 raft_consensus.cc:1067] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 [term 1 LEADER]: Signalling peer 3cda358d5d184a4383a9716334390022 to start an election
I20260807 03:38:00.712862 17367 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "80b452de6935470789989a31d5305c47"
dest_uuid: "3cda358d5d184a4383a9716334390022"
from {username='slave'} at 127.0.0.1:52424
I20260807 03:38:00.713333 17367 raft_consensus.cc:482] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:38:00.713635 17367 raft_consensus.cc:3037] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:00.718528 17367 raft_consensus.cc:504] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:00.720036 17367 leader_election.cc:288] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022 [CANDIDATE]: Term 2 election: Requested vote from peers 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887), 2224f937b1b347e2931b7383c84894f2 (127.15.198.131:41195)
I20260807 03:38:00.720886 17242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b452de6935470789989a31d5305c47" candidate_uuid: "3cda358d5d184a4383a9716334390022" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
I20260807 03:38:00.721156 17493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b452de6935470789989a31d5305c47" candidate_uuid: "3cda358d5d184a4383a9716334390022" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2224f937b1b347e2931b7383c84894f2"
I20260807 03:38:00.721516 17242 raft_consensus.cc:3037] T 80b452de6935470789989a31d5305c47 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:00.721735 17493 raft_consensus.cc:3032] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:00.722090 17493 raft_consensus.cc:729] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2224f937b1b347e2931b7383c84894f2, State: Running, Role: LEADER
I20260807 03:38:00.722721 17493 consensus_queue.cc:260] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:00.723605 17493 raft_consensus.cc:3037] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:00.727802 17493 raft_consensus.cc:2454] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cda358d5d184a4383a9716334390022 in term 2.
I20260807 03:38:00.728263 17242 raft_consensus.cc:2454] T 80b452de6935470789989a31d5305c47 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cda358d5d184a4383a9716334390022 in term 2.
I20260807 03:38:00.728708 17316 leader_election.cc:302] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2224f937b1b347e2931b7383c84894f2, 3cda358d5d184a4383a9716334390022; no voters:
I20260807 03:38:00.729420 17574 raft_consensus.cc:2781] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:38:00.729888 17574 raft_consensus.cc:686] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022 [term 2 LEADER]: Becoming Leader. State: Replica: 3cda358d5d184a4383a9716334390022, State: Running, Role: LEADER
I20260807 03:38:00.730655 17574 consensus_queue.cc:237] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:00.737818 17080 catalog_manager.cc:5495] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022 reported cstate change: term changed from 1 to 2, leader changed from 2224f937b1b347e2931b7383c84894f2 (127.15.198.131) to 3cda358d5d184a4383a9716334390022 (127.15.198.130). New cstate: current_term: 2 leader_uuid: "3cda358d5d184a4383a9716334390022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:00.841032 17574 raft_consensus.cc:979] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022: : Instructing follower 6c5dca27966a4b8cadb7cc0a8e0d1676 to start an election
I20260807 03:38:00.841430 17657 raft_consensus.cc:1067] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022 [term 1 LEADER]: Signalling peer 6c5dca27966a4b8cadb7cc0a8e0d1676 to start an election
I20260807 03:38:00.843931 17242 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a41a7b13d3284dcd8b9b616eeb93c578"
dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
from {username='slave'} at 127.0.0.1:60348
I20260807 03:38:00.844515 17242 raft_consensus.cc:482] T a41a7b13d3284dcd8b9b616eeb93c578 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:38:00.844911 17242 raft_consensus.cc:3037] T a41a7b13d3284dcd8b9b616eeb93c578 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:00.846019 17573 raft_consensus.cc:979] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676: : Instructing follower 3cda358d5d184a4383a9716334390022 to start an election
I20260807 03:38:00.846459 17607 raft_consensus.cc:1067] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 1 LEADER]: Signalling peer 3cda358d5d184a4383a9716334390022 to start an election
I20260807 03:38:00.848057 17367 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "efcd19aa11284c08ab3b4e1854a17dd9"
dest_uuid: "3cda358d5d184a4383a9716334390022"
from {username='slave'} at 127.0.0.1:52434
I20260807 03:38:00.848569 17367 raft_consensus.cc:482] T efcd19aa11284c08ab3b4e1854a17dd9 P 3cda358d5d184a4383a9716334390022 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:38:00.848868 17367 raft_consensus.cc:3037] T efcd19aa11284c08ab3b4e1854a17dd9 P 3cda358d5d184a4383a9716334390022 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:00.851316 17242 raft_consensus.cc:504] T a41a7b13d3284dcd8b9b616eeb93c578 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:00.852981 17242 leader_election.cc:288] T a41a7b13d3284dcd8b9b616eeb93c578 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [CANDIDATE]: Term 2 election: Requested vote from peers 3cda358d5d184a4383a9716334390022 (127.15.198.130:45465), 2224f937b1b347e2931b7383c84894f2 (127.15.198.131:41195)
I20260807 03:38:00.853691 17366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a41a7b13d3284dcd8b9b616eeb93c578" candidate_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3cda358d5d184a4383a9716334390022"
I20260807 03:38:00.854007 17493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a41a7b13d3284dcd8b9b616eeb93c578" candidate_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2224f937b1b347e2931b7383c84894f2"
I20260807 03:38:00.854267 17366 raft_consensus.cc:3032] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:00.854576 17493 raft_consensus.cc:3037] T a41a7b13d3284dcd8b9b616eeb93c578 P 2224f937b1b347e2931b7383c84894f2 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:00.854617 17366 raft_consensus.cc:729] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3cda358d5d184a4383a9716334390022, State: Running, Role: LEADER
I20260807 03:38:00.855082 17367 raft_consensus.cc:504] T efcd19aa11284c08ab3b4e1854a17dd9 P 3cda358d5d184a4383a9716334390022 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:00.855548 17366 consensus_queue.cc:260] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:00.856825 17366 raft_consensus.cc:3037] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:00.856981 17367 leader_election.cc:288] T efcd19aa11284c08ab3b4e1854a17dd9 P 3cda358d5d184a4383a9716334390022 [CANDIDATE]: Term 2 election: Requested vote from peers 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887), 2224f937b1b347e2931b7383c84894f2 (127.15.198.131:41195)
I20260807 03:38:00.857698 17242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcd19aa11284c08ab3b4e1854a17dd9" candidate_uuid: "3cda358d5d184a4383a9716334390022" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
I20260807 03:38:00.858286 17242 raft_consensus.cc:3032] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:00.858668 17242 raft_consensus.cc:729] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6c5dca27966a4b8cadb7cc0a8e0d1676, State: Running, Role: LEADER
I20260807 03:38:00.859030 17496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcd19aa11284c08ab3b4e1854a17dd9" candidate_uuid: "3cda358d5d184a4383a9716334390022" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2224f937b1b347e2931b7383c84894f2"
I20260807 03:38:00.859596 17496 raft_consensus.cc:3037] T efcd19aa11284c08ab3b4e1854a17dd9 P 2224f937b1b347e2931b7383c84894f2 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:00.859531 17242 consensus_queue.cc:260] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:00.861037 17242 raft_consensus.cc:3037] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:00.862393 17493 raft_consensus.cc:2454] T a41a7b13d3284dcd8b9b616eeb93c578 P 2224f937b1b347e2931b7383c84894f2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c5dca27966a4b8cadb7cc0a8e0d1676 in term 2.
I20260807 03:38:00.863407 17366 raft_consensus.cc:2454] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c5dca27966a4b8cadb7cc0a8e0d1676 in term 2.
I20260807 03:38:00.863451 17184 leader_election.cc:302] T a41a7b13d3284dcd8b9b616eeb93c578 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2224f937b1b347e2931b7383c84894f2, 6c5dca27966a4b8cadb7cc0a8e0d1676; no voters:
I20260807 03:38:00.864943 17607 raft_consensus.cc:2781] T a41a7b13d3284dcd8b9b616eeb93c578 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:38:00.865384 17607 raft_consensus.cc:686] T a41a7b13d3284dcd8b9b616eeb93c578 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 LEADER]: Becoming Leader. State: Replica: 6c5dca27966a4b8cadb7cc0a8e0d1676, State: Running, Role: LEADER
I20260807 03:38:00.865633 17496 raft_consensus.cc:2454] T efcd19aa11284c08ab3b4e1854a17dd9 P 2224f937b1b347e2931b7383c84894f2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cda358d5d184a4383a9716334390022 in term 2.
I20260807 03:38:00.866149 17607 consensus_queue.cc:237] T a41a7b13d3284dcd8b9b616eeb93c578 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:00.866719 17316 leader_election.cc:302] T efcd19aa11284c08ab3b4e1854a17dd9 P 3cda358d5d184a4383a9716334390022 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2224f937b1b347e2931b7383c84894f2, 3cda358d5d184a4383a9716334390022; no voters:
I20260807 03:38:00.867976 17242 raft_consensus.cc:2454] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cda358d5d184a4383a9716334390022 in term 2.
I20260807 03:38:00.869196 17574 raft_consensus.cc:2781] T efcd19aa11284c08ab3b4e1854a17dd9 P 3cda358d5d184a4383a9716334390022 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:38:00.869640 17574 raft_consensus.cc:686] T efcd19aa11284c08ab3b4e1854a17dd9 P 3cda358d5d184a4383a9716334390022 [term 2 LEADER]: Becoming Leader. State: Replica: 3cda358d5d184a4383a9716334390022, State: Running, Role: LEADER
I20260807 03:38:00.870458 17574 consensus_queue.cc:237] T efcd19aa11284c08ab3b4e1854a17dd9 P 3cda358d5d184a4383a9716334390022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:00.874679 17082 catalog_manager.cc:5495] T a41a7b13d3284dcd8b9b616eeb93c578 P 6c5dca27966a4b8cadb7cc0a8e0d1676 reported cstate change: term changed from 1 to 2, leader changed from 3cda358d5d184a4383a9716334390022 (127.15.198.130) to 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129). New cstate: current_term: 2 leader_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:00.879009 17080 catalog_manager.cc:5495] T efcd19aa11284c08ab3b4e1854a17dd9 P 3cda358d5d184a4383a9716334390022 reported cstate change: term changed from 1 to 2, leader changed from 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129) to 3cda358d5d184a4383a9716334390022 (127.15.198.130). New cstate: current_term: 2 leader_uuid: "3cda358d5d184a4383a9716334390022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:00.927971 17493 raft_consensus.cc:1261] T f99d56107b6a48db9611f1ce8282d575 P 2224f937b1b347e2931b7383c84894f2 [term 2 FOLLOWER]: Refusing update from remote peer 6c5dca27966a4b8cadb7cc0a8e0d1676: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:38:00.929435 17607 consensus_queue.cc:1035] T f99d56107b6a48db9611f1ce8282d575 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:00.938299 17366 raft_consensus.cc:1261] T f99d56107b6a48db9611f1ce8282d575 P 3cda358d5d184a4383a9716334390022 [term 2 FOLLOWER]: Refusing update from remote peer 6c5dca27966a4b8cadb7cc0a8e0d1676: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:38:00.939679 17607 consensus_queue.cc:1035] T f99d56107b6a48db9611f1ce8282d575 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:00.972868 17366 raft_consensus.cc:1261] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022 [term 2 FOLLOWER]: Refusing update from remote peer 6c5dca27966a4b8cadb7cc0a8e0d1676: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:38:00.974038 17607 consensus_queue.cc:1035] T 96451ce78a6f45479178c8d596430538 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:00.981184 17493 raft_consensus.cc:1261] T 96451ce78a6f45479178c8d596430538 P 2224f937b1b347e2931b7383c84894f2 [term 2 FOLLOWER]: Refusing update from remote peer 6c5dca27966a4b8cadb7cc0a8e0d1676: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:38:00.982491 17707 consensus_queue.cc:1035] T 96451ce78a6f45479178c8d596430538 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:01.162777 17493 raft_consensus.cc:1261] T 8f6f16d016d3437d8aeae4a64c42a856 P 2224f937b1b347e2931b7383c84894f2 [term 2 FOLLOWER]: Refusing update from remote peer 6c5dca27966a4b8cadb7cc0a8e0d1676: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:38:01.164003 17707 consensus_queue.cc:1035] T 8f6f16d016d3437d8aeae4a64c42a856 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:01.171387 17366 raft_consensus.cc:3037] T 8f6f16d016d3437d8aeae4a64c42a856 P 3cda358d5d184a4383a9716334390022 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:01.178627 17366 raft_consensus.cc:1261] T 8f6f16d016d3437d8aeae4a64c42a856 P 3cda358d5d184a4383a9716334390022 [term 2 FOLLOWER]: Refusing update from remote peer 6c5dca27966a4b8cadb7cc0a8e0d1676: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:38:01.179811 17607 consensus_queue.cc:1035] T 8f6f16d016d3437d8aeae4a64c42a856 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:01.226738 17242 raft_consensus.cc:1261] T 80b452de6935470789989a31d5305c47 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 FOLLOWER]: Refusing update from remote peer 3cda358d5d184a4383a9716334390022: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:38:01.227890 17574 consensus_queue.cc:1035] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:01.237562 17493 raft_consensus.cc:1261] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 [term 2 FOLLOWER]: Refusing update from remote peer 3cda358d5d184a4383a9716334390022: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:38:01.238777 17574 consensus_queue.cc:1035] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:01.332617 17493 raft_consensus.cc:1261] T a41a7b13d3284dcd8b9b616eeb93c578 P 2224f937b1b347e2931b7383c84894f2 [term 2 FOLLOWER]: Refusing update from remote peer 6c5dca27966a4b8cadb7cc0a8e0d1676: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:38:01.333737 17607 consensus_queue.cc:1035] T a41a7b13d3284dcd8b9b616eeb93c578 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:01.340606 17366 raft_consensus.cc:1261] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022 [term 2 FOLLOWER]: Refusing update from remote peer 6c5dca27966a4b8cadb7cc0a8e0d1676: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:38:01.342005 17607 consensus_queue.cc:1035] T a41a7b13d3284dcd8b9b616eeb93c578 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:01.412046 17493 raft_consensus.cc:1261] T efcd19aa11284c08ab3b4e1854a17dd9 P 2224f937b1b347e2931b7383c84894f2 [term 2 FOLLOWER]: Refusing update from remote peer 3cda358d5d184a4383a9716334390022: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:38:01.412209 17573 raft_consensus.cc:482] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:01.412612 17573 raft_consensus.cc:504] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:01.414211 17573 leader_election.cc:288] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3cda358d5d184a4383a9716334390022 (127.15.198.130:45465), 2224f937b1b347e2931b7383c84894f2 (127.15.198.131:41195)
I20260807 03:38:01.414729 17574 consensus_queue.cc:1035] T efcd19aa11284c08ab3b4e1854a17dd9 P 3cda358d5d184a4383a9716334390022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:01.414976 17366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f7b02979d648108ef56a2acf13fbf0" candidate_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cda358d5d184a4383a9716334390022" is_pre_election: true
I20260807 03:38:01.415082 17493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f7b02979d648108ef56a2acf13fbf0" candidate_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2224f937b1b347e2931b7383c84894f2" is_pre_election: true
I20260807 03:38:01.415659 17366 raft_consensus.cc:2454] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c5dca27966a4b8cadb7cc0a8e0d1676 in term 1.
I20260807 03:38:01.416621 17493 raft_consensus.cc:2396] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6c5dca27966a4b8cadb7cc0a8e0d1676 for term 3 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260807 03:38:01.416698 17184 leader_election.cc:302] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cda358d5d184a4383a9716334390022, 6c5dca27966a4b8cadb7cc0a8e0d1676; no voters:
I20260807 03:38:01.417641 17573 raft_consensus.cc:2781] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:01.418565 17573 raft_consensus.cc:482] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:01.419059 17573 raft_consensus.cc:3037] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:01.419588 17242 raft_consensus.cc:1261] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 FOLLOWER]: Refusing update from remote peer 3cda358d5d184a4383a9716334390022: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:38:01.421308 17658 consensus_queue.cc:1035] T efcd19aa11284c08ab3b4e1854a17dd9 P 3cda358d5d184a4383a9716334390022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:01.431236 17573 raft_consensus.cc:504] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:01.433938 17573 leader_election.cc:288] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [CANDIDATE]: Term 3 election: Requested vote from peers 3cda358d5d184a4383a9716334390022 (127.15.198.130:45465), 2224f937b1b347e2931b7383c84894f2 (127.15.198.131:41195)
I20260807 03:38:01.436400 17367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f7b02979d648108ef56a2acf13fbf0" candidate_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cda358d5d184a4383a9716334390022"
I20260807 03:38:01.436975 17367 raft_consensus.cc:3037] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022 [term 1 FOLLOWER]: Advancing to term 3
I20260807 03:38:01.440366 17496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f7b02979d648108ef56a2acf13fbf0" candidate_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2224f937b1b347e2931b7383c84894f2"
I20260807 03:38:01.440840 17496 raft_consensus.cc:3037] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:01.442798 17367 raft_consensus.cc:2454] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c5dca27966a4b8cadb7cc0a8e0d1676 in term 3.
I20260807 03:38:01.443768 17184 leader_election.cc:302] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cda358d5d184a4383a9716334390022, 6c5dca27966a4b8cadb7cc0a8e0d1676; no voters:
I20260807 03:38:01.444502 17573 raft_consensus.cc:2781] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:38:01.444955 17573 raft_consensus.cc:686] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 3 LEADER]: Becoming Leader. State: Replica: 6c5dca27966a4b8cadb7cc0a8e0d1676, State: Running, Role: LEADER
I20260807 03:38:01.446030 17573 consensus_queue.cc:237] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:01.447893 17496 raft_consensus.cc:2396] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6c5dca27966a4b8cadb7cc0a8e0d1676 for term 3 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260807 03:38:01.454219 17082 catalog_manager.cc:5495] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 reported cstate change: term changed from 1 to 3, leader changed from 2224f937b1b347e2931b7383c84894f2 (127.15.198.131) to 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129). New cstate: current_term: 3 leader_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:01.894779 17496 raft_consensus.cc:1261] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 3 FOLLOWER]: Refusing update from remote peer 6c5dca27966a4b8cadb7cc0a8e0d1676: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 1. (term mismatch)
I20260807 03:38:01.895246 17496 pending_rounds.cc:76] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2: Aborting all ops after (but not including) 0
I20260807 03:38:01.895478 17496 pending_rounds.cc:97] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260807 03:38:01.895809 17496 raft_consensus.cc:2866] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 03:38:01.896986 17573 consensus_queue.cc:1035] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:01.897979 17611 raft_consensus.cc:482] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:01.898458 17611 raft_consensus.cc:504] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:01.900220 17611 leader_election.cc:288] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887), 3cda358d5d184a4383a9716334390022 (127.15.198.130:45465)
I20260807 03:38:01.900859 17242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f7b02979d648108ef56a2acf13fbf0" candidate_uuid: "2224f937b1b347e2931b7383c84894f2" candidate_term: 4 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" is_pre_election: true
I20260807 03:38:01.901842 17367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f7b02979d648108ef56a2acf13fbf0" candidate_uuid: "2224f937b1b347e2931b7383c84894f2" candidate_term: 4 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cda358d5d184a4383a9716334390022" is_pre_election: true
I20260807 03:38:01.902412 17367 raft_consensus.cc:2454] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2224f937b1b347e2931b7383c84894f2 in term 3.
I20260807 03:38:01.903482 17443 leader_election.cc:302] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2224f937b1b347e2931b7383c84894f2, 3cda358d5d184a4383a9716334390022; no voters: 6c5dca27966a4b8cadb7cc0a8e0d1676
I20260807 03:38:01.904342 17611 raft_consensus.cc:2781] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:38:01.904657 17611 raft_consensus.cc:482] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6c5dca27966a4b8cadb7cc0a8e0d1676)
I20260807 03:38:01.904994 17611 raft_consensus.cc:3037] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:38:01.907100 17367 raft_consensus.cc:1261] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022 [term 3 FOLLOWER]: Refusing update from remote peer 6c5dca27966a4b8cadb7cc0a8e0d1676: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 1. (index mismatch)
I20260807 03:38:01.908562 17573 consensus_queue.cc:1035] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:01.911983 17611 raft_consensus.cc:504] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:01.914551 17611 leader_election.cc:288] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [CANDIDATE]: Term 4 election: Requested vote from peers 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887), 3cda358d5d184a4383a9716334390022 (127.15.198.130:45465)
I20260807 03:38:01.915160 17366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f7b02979d648108ef56a2acf13fbf0" candidate_uuid: "2224f937b1b347e2931b7383c84894f2" candidate_term: 4 candidate_status { last_received { term: 3 index: 1 } } ignore_live_leader: false dest_uuid: "3cda358d5d184a4383a9716334390022"
I20260807 03:38:01.916978 17496 raft_consensus.cc:1226] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 4 FOLLOWER]: Rejecting Update request from peer 6c5dca27966a4b8cadb7cc0a8e0d1676 for earlier term 3. Current term is 4. Ops: []
I20260807 03:38:01.917424 17242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f7b02979d648108ef56a2acf13fbf0" candidate_uuid: "2224f937b1b347e2931b7383c84894f2" candidate_term: 4 candidate_status { last_received { term: 3 index: 1 } } ignore_live_leader: false dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
I20260807 03:38:01.919064 17440 leader_election.cc:302] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2224f937b1b347e2931b7383c84894f2; no voters: 3cda358d5d184a4383a9716334390022, 6c5dca27966a4b8cadb7cc0a8e0d1676
I20260807 03:38:01.920131 17611 raft_consensus.cc:2726] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 03:38:01.922379 17707 consensus_queue.cc:1046] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 }, Status: INVALID_TERM, Last received: 3.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:01.923092 17607 raft_consensus.cc:3032] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:38:01.923410 17607 raft_consensus.cc:729] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6c5dca27966a4b8cadb7cc0a8e0d1676, State: Running, Role: LEADER
I20260807 03:38:01.924021 17607 consensus_queue.cc:260] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.1, Last appended by leader: 1, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:01.925189 17607 raft_consensus.cc:3037] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:38:03.412158 17574 raft_consensus.cc:482] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6c5dca27966a4b8cadb7cc0a8e0d1676)
I20260807 03:38:03.412676 17574 raft_consensus.cc:504] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:03.414445 17574 leader_election.cc:288] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887), 2224f937b1b347e2931b7383c84894f2 (127.15.198.131:41195)
I20260807 03:38:03.415223 17246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f7b02979d648108ef56a2acf13fbf0" candidate_uuid: "3cda358d5d184a4383a9716334390022" candidate_term: 4 candidate_status { last_received { term: 3 index: 1 } } ignore_live_leader: false dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" is_pre_election: true
I20260807 03:38:03.415544 17496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f7b02979d648108ef56a2acf13fbf0" candidate_uuid: "3cda358d5d184a4383a9716334390022" candidate_term: 4 candidate_status { last_received { term: 3 index: 1 } } ignore_live_leader: false dest_uuid: "2224f937b1b347e2931b7383c84894f2" is_pre_election: true
I20260807 03:38:03.415930 17246 raft_consensus.cc:2454] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cda358d5d184a4383a9716334390022 in term 4.
I20260807 03:38:03.416376 17496 raft_consensus.cc:2379] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3cda358d5d184a4383a9716334390022 in current term 4: Already voted for candidate 2224f937b1b347e2931b7383c84894f2 in this term.
I20260807 03:38:03.417088 17313 leader_election.cc:302] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cda358d5d184a4383a9716334390022, 6c5dca27966a4b8cadb7cc0a8e0d1676; no voters:
I20260807 03:38:03.417778 17574 raft_consensus.cc:2781] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:38:03.418113 17574 raft_consensus.cc:482] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6c5dca27966a4b8cadb7cc0a8e0d1676)
I20260807 03:38:03.418424 17574 raft_consensus.cc:3037] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:38:03.422956 17574 raft_consensus.cc:504] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:03.424319 17574 leader_election.cc:288] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022 [CANDIDATE]: Term 4 election: Requested vote from peers 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887), 2224f937b1b347e2931b7383c84894f2 (127.15.198.131:41195)
I20260807 03:38:03.424955 17246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f7b02979d648108ef56a2acf13fbf0" candidate_uuid: "3cda358d5d184a4383a9716334390022" candidate_term: 4 candidate_status { last_received { term: 3 index: 1 } } ignore_live_leader: false dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
I20260807 03:38:03.425223 17496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f7b02979d648108ef56a2acf13fbf0" candidate_uuid: "3cda358d5d184a4383a9716334390022" candidate_term: 4 candidate_status { last_received { term: 3 index: 1 } } ignore_live_leader: false dest_uuid: "2224f937b1b347e2931b7383c84894f2"
I20260807 03:38:03.425848 17496 raft_consensus.cc:2379] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3cda358d5d184a4383a9716334390022 in current term 4: Already voted for candidate 2224f937b1b347e2931b7383c84894f2 in this term.
I20260807 03:38:03.429328 17246 raft_consensus.cc:2454] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cda358d5d184a4383a9716334390022 in term 4.
I20260807 03:38:03.430220 17313 leader_election.cc:302] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3cda358d5d184a4383a9716334390022, 6c5dca27966a4b8cadb7cc0a8e0d1676; no voters: 2224f937b1b347e2931b7383c84894f2
I20260807 03:38:03.430822 17574 raft_consensus.cc:2781] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 03:38:03.431284 17574 raft_consensus.cc:686] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022 [term 4 LEADER]: Becoming Leader. State: Replica: 3cda358d5d184a4383a9716334390022, State: Running, Role: LEADER
I20260807 03:38:03.431919 17574 consensus_queue.cc:237] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:03.437546 17082 catalog_manager.cc:5495] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022 reported cstate change: term changed from 3 to 4, leader changed from 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129) to 3cda358d5d184a4383a9716334390022 (127.15.198.130). New cstate: current_term: 4 leader_uuid: "3cda358d5d184a4383a9716334390022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:03.473625 16154 auto_leader_rebalancer-test.cc:339] The leader distribution is
I20260807 03:38:03.474213 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:38:03.493166 17246 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "96451ce78a6f45479178c8d596430538"
dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
mode: GRACEFUL
new_leader_uuid: "2224f937b1b347e2931b7383c84894f2"
from {username='slave'} at 127.0.0.1:60390
I20260807 03:38:03.493672 17246 raft_consensus.cc:595] T 96451ce78a6f45479178c8d596430538 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 LEADER]: Received request to transfer leadership to TS 2224f937b1b347e2931b7383c84894f2
I20260807 03:38:03.504688 17367 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "efcd19aa11284c08ab3b4e1854a17dd9"
dest_uuid: "3cda358d5d184a4383a9716334390022"
mode: GRACEFUL
new_leader_uuid: "2224f937b1b347e2931b7383c84894f2"
from {username='slave'} at 127.0.0.1:52442
I20260807 03:38:03.505160 17367 raft_consensus.cc:595] T efcd19aa11284c08ab3b4e1854a17dd9 P 3cda358d5d184a4383a9716334390022 [term 2 LEADER]: Received request to transfer leadership to TS 2224f937b1b347e2931b7383c84894f2
I20260807 03:38:03.506104 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260807 03:38:03.506552 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:38:03.601925 17611 raft_consensus.cc:482] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:03.602453 17611 raft_consensus.cc:504] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:03.604346 17611 leader_election.cc:288] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887), 3cda358d5d184a4383a9716334390022 (127.15.198.130:45465)
I20260807 03:38:03.605131 17246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f7b02979d648108ef56a2acf13fbf0" candidate_uuid: "2224f937b1b347e2931b7383c84894f2" candidate_term: 5 candidate_status { last_received { term: 3 index: 1 } } ignore_live_leader: false dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" is_pre_election: true
I20260807 03:38:03.605378 17367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f7b02979d648108ef56a2acf13fbf0" candidate_uuid: "2224f937b1b347e2931b7383c84894f2" candidate_term: 5 candidate_status { last_received { term: 3 index: 1 } } ignore_live_leader: false dest_uuid: "3cda358d5d184a4383a9716334390022" is_pre_election: true
I20260807 03:38:03.605785 17246 raft_consensus.cc:2454] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2224f937b1b347e2931b7383c84894f2 in term 4.
I20260807 03:38:03.606715 17440 leader_election.cc:302] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2224f937b1b347e2931b7383c84894f2, 6c5dca27966a4b8cadb7cc0a8e0d1676; no voters:
I20260807 03:38:03.607406 17611 raft_consensus.cc:2781] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:38:03.607712 17611 raft_consensus.cc:482] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:03.607992 17611 raft_consensus.cc:3037] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:38:03.613487 17611 raft_consensus.cc:504] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:03.615023 17611 leader_election.cc:288] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [CANDIDATE]: Term 5 election: Requested vote from peers 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887), 3cda358d5d184a4383a9716334390022 (127.15.198.130:45465)
I20260807 03:38:03.615751 17246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f7b02979d648108ef56a2acf13fbf0" candidate_uuid: "2224f937b1b347e2931b7383c84894f2" candidate_term: 5 candidate_status { last_received { term: 3 index: 1 } } ignore_live_leader: false dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
I20260807 03:38:03.616333 17246 raft_consensus.cc:3037] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:38:03.616585 17367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f7b02979d648108ef56a2acf13fbf0" candidate_uuid: "2224f937b1b347e2931b7383c84894f2" candidate_term: 5 candidate_status { last_received { term: 3 index: 1 } } ignore_live_leader: false dest_uuid: "3cda358d5d184a4383a9716334390022"
I20260807 03:38:03.620743 17246 raft_consensus.cc:2454] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2224f937b1b347e2931b7383c84894f2 in term 5.
I20260807 03:38:03.621565 17440 leader_election.cc:302] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2224f937b1b347e2931b7383c84894f2, 6c5dca27966a4b8cadb7cc0a8e0d1676; no voters: 3cda358d5d184a4383a9716334390022
I20260807 03:38:03.622167 17611 raft_consensus.cc:2781] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 03:38:03.623138 17611 raft_consensus.cc:686] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 5 LEADER]: Becoming Leader. State: Replica: 2224f937b1b347e2931b7383c84894f2, State: Running, Role: LEADER
I20260807 03:38:03.623879 17611 consensus_queue.cc:237] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 3.1, Last appended by leader: 1, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:03.630003 17082 catalog_manager.cc:5495] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 reported cstate change: term changed from 4 to 5, leader changed from 3cda358d5d184a4383a9716334390022 (127.15.198.130) to 2224f937b1b347e2931b7383c84894f2 (127.15.198.131). New cstate: current_term: 5 leader_uuid: "2224f937b1b347e2931b7383c84894f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } health_report { overall_health: HEALTHY } } }
I20260807 03:38:03.864432 17496 raft_consensus.cc:1226] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 5 LEADER]: Rejecting Update request from peer 3cda358d5d184a4383a9716334390022 for earlier term 4. Current term is 5. Ops: []
I20260807 03:38:03.865600 17574 consensus_queue.cc:1046] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:03.866140 17574 consensus_queue.cc:1225] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022 [LEADER]: Peer 2224f937b1b347e2931b7383c84894f2 log is divergent from this leader: its last log entry 5.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20260807 03:38:03.866473 17721 raft_consensus.cc:3032] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 03:38:03.866722 17721 raft_consensus.cc:729] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3cda358d5d184a4383a9716334390022, State: Running, Role: LEADER
I20260807 03:38:03.867278 17721 consensus_queue.cc:260] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:03.868146 17721 raft_consensus.cc:3037] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:38:03.957713 17607 raft_consensus.cc:979] T 96451ce78a6f45479178c8d596430538 P 6c5dca27966a4b8cadb7cc0a8e0d1676: : Instructing follower 2224f937b1b347e2931b7383c84894f2 to start an election
I20260807 03:38:03.958150 17607 raft_consensus.cc:1067] T 96451ce78a6f45479178c8d596430538 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 LEADER]: Signalling peer 2224f937b1b347e2931b7383c84894f2 to start an election
I20260807 03:38:03.959390 17496 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "96451ce78a6f45479178c8d596430538"
dest_uuid: "2224f937b1b347e2931b7383c84894f2"
from {username='slave'} at 127.0.0.1:48312
I20260807 03:38:03.959854 17496 raft_consensus.cc:482] T 96451ce78a6f45479178c8d596430538 P 2224f937b1b347e2931b7383c84894f2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:38:03.960172 17496 raft_consensus.cc:3037] T 96451ce78a6f45479178c8d596430538 P 2224f937b1b347e2931b7383c84894f2 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:03.964517 17496 raft_consensus.cc:504] T 96451ce78a6f45479178c8d596430538 P 2224f937b1b347e2931b7383c84894f2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } }
I20260807 03:38:03.965909 17496 leader_election.cc:288] T 96451ce78a6f45479178c8d596430538 P 2224f937b1b347e2931b7383c84894f2 [CANDIDATE]: Term 3 election: Requested vote from peers 3cda358d5d184a4383a9716334390022 (127.15.198.130:45465), 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887)
I20260807 03:38:03.966890 17246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96451ce78a6f45479178c8d596430538" candidate_uuid: "2224f937b1b347e2931b7383c84894f2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
I20260807 03:38:03.966912 17367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96451ce78a6f45479178c8d596430538" candidate_uuid: "2224f937b1b347e2931b7383c84894f2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3cda358d5d184a4383a9716334390022"
I20260807 03:38:03.967429 17246 raft_consensus.cc:3032] T 96451ce78a6f45479178c8d596430538 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 03:38:03.967540 17367 raft_consensus.cc:3037] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:03.967732 17246 raft_consensus.cc:729] T 96451ce78a6f45479178c8d596430538 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6c5dca27966a4b8cadb7cc0a8e0d1676, State: Running, Role: LEADER
I20260807 03:38:03.968505 17246 consensus_queue.cc:260] T 96451ce78a6f45479178c8d596430538 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } }
I20260807 03:38:03.969398 17246 raft_consensus.cc:3037] T 96451ce78a6f45479178c8d596430538 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:03.972256 17367 raft_consensus.cc:2454] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2224f937b1b347e2931b7383c84894f2 in term 3.
I20260807 03:38:03.973100 17443 leader_election.cc:302] T 96451ce78a6f45479178c8d596430538 P 2224f937b1b347e2931b7383c84894f2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2224f937b1b347e2931b7383c84894f2, 3cda358d5d184a4383a9716334390022; no voters:
I20260807 03:38:03.973733 17611 raft_consensus.cc:2781] T 96451ce78a6f45479178c8d596430538 P 2224f937b1b347e2931b7383c84894f2 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:38:03.973949 17246 raft_consensus.cc:2454] T 96451ce78a6f45479178c8d596430538 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2224f937b1b347e2931b7383c84894f2 in term 3.
I20260807 03:38:03.974213 17611 raft_consensus.cc:686] T 96451ce78a6f45479178c8d596430538 P 2224f937b1b347e2931b7383c84894f2 [term 3 LEADER]: Becoming Leader. State: Replica: 2224f937b1b347e2931b7383c84894f2, State: Running, Role: LEADER
I20260807 03:38:03.974967 17611 consensus_queue.cc:237] T 96451ce78a6f45479178c8d596430538 P 2224f937b1b347e2931b7383c84894f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } }
I20260807 03:38:03.982021 17082 catalog_manager.cc:5495] T 96451ce78a6f45479178c8d596430538 P 2224f937b1b347e2931b7383c84894f2 reported cstate change: term changed from 2 to 3, leader changed from 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129) to 2224f937b1b347e2931b7383c84894f2 (127.15.198.131). New cstate: current_term: 3 leader_uuid: "2224f937b1b347e2931b7383c84894f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:03.982988 17721 raft_consensus.cc:979] T efcd19aa11284c08ab3b4e1854a17dd9 P 3cda358d5d184a4383a9716334390022: : Instructing follower 2224f937b1b347e2931b7383c84894f2 to start an election
I20260807 03:38:03.983435 17739 raft_consensus.cc:1067] T efcd19aa11284c08ab3b4e1854a17dd9 P 3cda358d5d184a4383a9716334390022 [term 2 LEADER]: Signalling peer 2224f937b1b347e2931b7383c84894f2 to start an election
I20260807 03:38:03.984958 17496 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "efcd19aa11284c08ab3b4e1854a17dd9"
dest_uuid: "2224f937b1b347e2931b7383c84894f2"
from {username='slave'} at 127.0.0.1:48304
I20260807 03:38:03.985466 17496 raft_consensus.cc:482] T efcd19aa11284c08ab3b4e1854a17dd9 P 2224f937b1b347e2931b7383c84894f2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:38:03.985791 17496 raft_consensus.cc:3037] T efcd19aa11284c08ab3b4e1854a17dd9 P 2224f937b1b347e2931b7383c84894f2 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:03.990900 17496 raft_consensus.cc:504] T efcd19aa11284c08ab3b4e1854a17dd9 P 2224f937b1b347e2931b7383c84894f2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:03.992172 17496 leader_election.cc:288] T efcd19aa11284c08ab3b4e1854a17dd9 P 2224f937b1b347e2931b7383c84894f2 [CANDIDATE]: Term 3 election: Requested vote from peers 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887), 3cda358d5d184a4383a9716334390022 (127.15.198.130:45465)
I20260807 03:38:03.992924 17246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcd19aa11284c08ab3b4e1854a17dd9" candidate_uuid: "2224f937b1b347e2931b7383c84894f2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
I20260807 03:38:03.993364 17246 raft_consensus.cc:3037] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:03.993280 17367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcd19aa11284c08ab3b4e1854a17dd9" candidate_uuid: "2224f937b1b347e2931b7383c84894f2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3cda358d5d184a4383a9716334390022"
I20260807 03:38:03.993801 17367 raft_consensus.cc:3032] T efcd19aa11284c08ab3b4e1854a17dd9 P 3cda358d5d184a4383a9716334390022 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 03:38:03.994004 17367 raft_consensus.cc:729] T efcd19aa11284c08ab3b4e1854a17dd9 P 3cda358d5d184a4383a9716334390022 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3cda358d5d184a4383a9716334390022, State: Running, Role: LEADER
I20260807 03:38:03.994630 17367 consensus_queue.cc:260] T efcd19aa11284c08ab3b4e1854a17dd9 P 3cda358d5d184a4383a9716334390022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:03.995594 17367 raft_consensus.cc:3037] T efcd19aa11284c08ab3b4e1854a17dd9 P 3cda358d5d184a4383a9716334390022 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:03.997813 17246 raft_consensus.cc:2454] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2224f937b1b347e2931b7383c84894f2 in term 3.
I20260807 03:38:03.998813 17440 leader_election.cc:302] T efcd19aa11284c08ab3b4e1854a17dd9 P 2224f937b1b347e2931b7383c84894f2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2224f937b1b347e2931b7383c84894f2, 6c5dca27966a4b8cadb7cc0a8e0d1676; no voters:
I20260807 03:38:03.999670 17611 raft_consensus.cc:2781] T efcd19aa11284c08ab3b4e1854a17dd9 P 2224f937b1b347e2931b7383c84894f2 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:38:04.000274 17611 raft_consensus.cc:686] T efcd19aa11284c08ab3b4e1854a17dd9 P 2224f937b1b347e2931b7383c84894f2 [term 3 LEADER]: Becoming Leader. State: Replica: 2224f937b1b347e2931b7383c84894f2, State: Running, Role: LEADER
I20260807 03:38:04.001307 17367 raft_consensus.cc:2454] T efcd19aa11284c08ab3b4e1854a17dd9 P 3cda358d5d184a4383a9716334390022 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2224f937b1b347e2931b7383c84894f2 in term 3.
I20260807 03:38:04.001144 17611 consensus_queue.cc:237] T efcd19aa11284c08ab3b4e1854a17dd9 P 2224f937b1b347e2931b7383c84894f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:04.008289 17082 catalog_manager.cc:5495] T efcd19aa11284c08ab3b4e1854a17dd9 P 2224f937b1b347e2931b7383c84894f2 reported cstate change: term changed from 2 to 3, leader changed from 3cda358d5d184a4383a9716334390022 (127.15.198.130) to 2224f937b1b347e2931b7383c84894f2 (127.15.198.131). New cstate: current_term: 3 leader_uuid: "2224f937b1b347e2931b7383c84894f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } health_report { overall_health: HEALTHY } } }
I20260807 03:38:04.057077 17246 raft_consensus.cc:1261] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 5 FOLLOWER]: Refusing update from remote peer 2224f937b1b347e2931b7383c84894f2: Log matching property violated. Preceding OpId in replica: term: 3 index: 1. Preceding OpId from leader: term: 5 index: 2. (index mismatch)
I20260807 03:38:04.058166 17611 consensus_queue.cc:1035] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:04.068698 17367 raft_consensus.cc:1261] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022 [term 5 FOLLOWER]: Refusing update from remote peer 2224f937b1b347e2931b7383c84894f2: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 2. (term mismatch)
I20260807 03:38:04.069069 17367 pending_rounds.cc:76] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022: Aborting all ops after (but not including) 1
I20260807 03:38:04.069298 17367 pending_rounds.cc:97] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022: Aborting uncommitted NO_OP operation due to leader change: 4.2
I20260807 03:38:04.069537 17367 raft_consensus.cc:2866] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 03:38:04.071226 17747 consensus_queue.cc:1035] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:04.411711 17246 raft_consensus.cc:1261] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 3 FOLLOWER]: Refusing update from remote peer 2224f937b1b347e2931b7383c84894f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 03:38:04.413292 17747 consensus_queue.cc:1035] T efcd19aa11284c08ab3b4e1854a17dd9 P 2224f937b1b347e2931b7383c84894f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:38:04.423504 17367 raft_consensus.cc:1261] T efcd19aa11284c08ab3b4e1854a17dd9 P 3cda358d5d184a4383a9716334390022 [term 3 FOLLOWER]: Refusing update from remote peer 2224f937b1b347e2931b7383c84894f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 03:38:04.425249 17752 consensus_queue.cc:1035] T efcd19aa11284c08ab3b4e1854a17dd9 P 2224f937b1b347e2931b7383c84894f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 03:38:04.432931 17366 raft_consensus.cc:1261] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022 [term 3 FOLLOWER]: Refusing update from remote peer 2224f937b1b347e2931b7383c84894f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 03:38:04.434154 17752 consensus_queue.cc:1035] T 96451ce78a6f45479178c8d596430538 P 2224f937b1b347e2931b7383c84894f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:38:04.446632 17246 raft_consensus.cc:1261] T 96451ce78a6f45479178c8d596430538 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 3 FOLLOWER]: Refusing update from remote peer 2224f937b1b347e2931b7383c84894f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 03:38:04.448794 17752 consensus_queue.cc:1035] T 96451ce78a6f45479178c8d596430538 P 2224f937b1b347e2931b7383c84894f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:38:04.507391 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:38:04.513162 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 03:38:04.513656 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:38:05.514477 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:38:05.520432 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 03:38:05.520804 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:38:06.521739 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:38:06.527685 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 03:38:06.528066 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:38:07.528895 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:38:07.534147 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 03:38:07.534565 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:38:08.535495 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:38:08.540580 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 03:38:08.540980 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:38:09.541744 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:38:09.547497 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 03:38:09.547847 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:38:10.548693 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:38:10.553551 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 03:38:10.553929 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:38:11.554641 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:38:11.559559 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 03:38:11.560001 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:38:12.560848 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:38:12.565838 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 03:38:12.566257 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:38:13.593498 16154 auto_leader_rebalancer-test.cc:358] The leader distribution is
I20260807 03:38:13.597971 17496 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "96451ce78a6f45479178c8d596430538"
dest_uuid: "2224f937b1b347e2931b7383c84894f2"
mode: GRACEFUL
new_leader_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
from {username='slave'} at 127.0.0.1:48322
I20260807 03:38:13.598572 17496 raft_consensus.cc:595] T 96451ce78a6f45479178c8d596430538 P 2224f937b1b347e2931b7383c84894f2 [term 3 LEADER]: Received request to transfer leadership to TS 6c5dca27966a4b8cadb7cc0a8e0d1676
I20260807 03:38:13.602685 17496 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e3f7b02979d648108ef56a2acf13fbf0"
dest_uuid: "2224f937b1b347e2931b7383c84894f2"
mode: GRACEFUL
new_leader_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
from {username='slave'} at 127.0.0.1:48322
I20260807 03:38:13.603091 17496 raft_consensus.cc:595] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 5 LEADER]: Received request to transfer leadership to TS 6c5dca27966a4b8cadb7cc0a8e0d1676
I20260807 03:38:13.614420 17366 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "80b452de6935470789989a31d5305c47"
dest_uuid: "3cda358d5d184a4383a9716334390022"
mode: GRACEFUL
new_leader_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
from {username='slave'} at 127.0.0.1:52436
I20260807 03:38:13.614936 17366 raft_consensus.cc:595] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022 [term 2 LEADER]: Received request to transfer leadership to TS 6c5dca27966a4b8cadb7cc0a8e0d1676
I20260807 03:38:13.619257 17496 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "efcd19aa11284c08ab3b4e1854a17dd9"
dest_uuid: "2224f937b1b347e2931b7383c84894f2"
mode: GRACEFUL
new_leader_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
from {username='slave'} at 127.0.0.1:48322
I20260807 03:38:13.619669 17496 raft_consensus.cc:595] T efcd19aa11284c08ab3b4e1854a17dd9 P 2224f937b1b347e2931b7383c84894f2 [term 3 LEADER]: Received request to transfer leadership to TS 6c5dca27966a4b8cadb7cc0a8e0d1676
I20260807 03:38:13.623545 17366 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "64fc83d22ad945ad890e67f9763fe413"
dest_uuid: "3cda358d5d184a4383a9716334390022"
mode: GRACEFUL
new_leader_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
from {username='slave'} at 127.0.0.1:52436
I20260807 03:38:13.623970 17366 raft_consensus.cc:595] T 64fc83d22ad945ad890e67f9763fe413 P 3cda358d5d184a4383a9716334390022 [term 1 LEADER]: Received request to transfer leadership to TS 6c5dca27966a4b8cadb7cc0a8e0d1676
I20260807 03:38:13.679745 17764 raft_consensus.cc:979] T efcd19aa11284c08ab3b4e1854a17dd9 P 2224f937b1b347e2931b7383c84894f2: : Instructing follower 6c5dca27966a4b8cadb7cc0a8e0d1676 to start an election
I20260807 03:38:13.680263 17766 raft_consensus.cc:1067] T efcd19aa11284c08ab3b4e1854a17dd9 P 2224f937b1b347e2931b7383c84894f2 [term 3 LEADER]: Signalling peer 6c5dca27966a4b8cadb7cc0a8e0d1676 to start an election
I20260807 03:38:13.681684 17246 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "efcd19aa11284c08ab3b4e1854a17dd9"
dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
from {username='slave'} at 127.0.0.1:60360
I20260807 03:38:13.682201 17246 raft_consensus.cc:482] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:38:13.682626 17246 raft_consensus.cc:3037] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:38:13.689397 17246 raft_consensus.cc:504] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:13.691005 17246 leader_election.cc:288] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [CANDIDATE]: Term 4 election: Requested vote from peers 3cda358d5d184a4383a9716334390022 (127.15.198.130:45465), 2224f937b1b347e2931b7383c84894f2 (127.15.198.131:41195)
I20260807 03:38:13.691644 17366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcd19aa11284c08ab3b4e1854a17dd9" candidate_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "3cda358d5d184a4383a9716334390022"
I20260807 03:38:13.691982 17496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcd19aa11284c08ab3b4e1854a17dd9" candidate_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "2224f937b1b347e2931b7383c84894f2"
I20260807 03:38:13.692487 17366 raft_consensus.cc:3037] T efcd19aa11284c08ab3b4e1854a17dd9 P 3cda358d5d184a4383a9716334390022 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:38:13.692662 17496 raft_consensus.cc:3032] T efcd19aa11284c08ab3b4e1854a17dd9 P 2224f937b1b347e2931b7383c84894f2 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:38:13.693020 17496 raft_consensus.cc:729] T efcd19aa11284c08ab3b4e1854a17dd9 P 2224f937b1b347e2931b7383c84894f2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2224f937b1b347e2931b7383c84894f2, State: Running, Role: LEADER
I20260807 03:38:13.693614 17496 consensus_queue.cc:260] T efcd19aa11284c08ab3b4e1854a17dd9 P 2224f937b1b347e2931b7383c84894f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:13.694721 17496 raft_consensus.cc:3037] T efcd19aa11284c08ab3b4e1854a17dd9 P 2224f937b1b347e2931b7383c84894f2 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:38:13.697196 17366 raft_consensus.cc:2454] T efcd19aa11284c08ab3b4e1854a17dd9 P 3cda358d5d184a4383a9716334390022 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c5dca27966a4b8cadb7cc0a8e0d1676 in term 4.
I20260807 03:38:13.698192 17184 leader_election.cc:302] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cda358d5d184a4383a9716334390022, 6c5dca27966a4b8cadb7cc0a8e0d1676; no voters:
I20260807 03:38:13.698961 17763 raft_consensus.cc:2781] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 03:38:13.699906 17763 raft_consensus.cc:686] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 4 LEADER]: Becoming Leader. State: Replica: 6c5dca27966a4b8cadb7cc0a8e0d1676, State: Running, Role: LEADER
I20260807 03:38:13.700410 17496 raft_consensus.cc:2454] T efcd19aa11284c08ab3b4e1854a17dd9 P 2224f937b1b347e2931b7383c84894f2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c5dca27966a4b8cadb7cc0a8e0d1676 in term 4.
I20260807 03:38:13.700845 17763 consensus_queue.cc:237] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:13.707638 17082 catalog_manager.cc:5495] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676 reported cstate change: term changed from 3 to 4, leader changed from 2224f937b1b347e2931b7383c84894f2 (127.15.198.131) to 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129). New cstate: current_term: 4 leader_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:13.745991 17778 raft_consensus.cc:979] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022: : Instructing follower 6c5dca27966a4b8cadb7cc0a8e0d1676 to start an election
I20260807 03:38:13.746543 17765 raft_consensus.cc:1067] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022 [term 2 LEADER]: Signalling peer 6c5dca27966a4b8cadb7cc0a8e0d1676 to start an election
I20260807 03:38:13.747820 17246 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "80b452de6935470789989a31d5305c47"
dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
from {username='slave'} at 127.0.0.1:60348
I20260807 03:38:13.748263 17246 raft_consensus.cc:482] T 80b452de6935470789989a31d5305c47 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:38:13.748540 17246 raft_consensus.cc:3037] T 80b452de6935470789989a31d5305c47 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:13.752801 17246 raft_consensus.cc:504] T 80b452de6935470789989a31d5305c47 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:13.754105 17246 leader_election.cc:288] T 80b452de6935470789989a31d5305c47 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [CANDIDATE]: Term 3 election: Requested vote from peers 3cda358d5d184a4383a9716334390022 (127.15.198.130:45465), 2224f937b1b347e2931b7383c84894f2 (127.15.198.131:41195)
I20260807 03:38:13.754698 17366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b452de6935470789989a31d5305c47" candidate_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3cda358d5d184a4383a9716334390022"
I20260807 03:38:13.755187 17366 raft_consensus.cc:3032] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 03:38:13.755095 17496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b452de6935470789989a31d5305c47" candidate_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2224f937b1b347e2931b7383c84894f2"
I20260807 03:38:13.755492 17366 raft_consensus.cc:729] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3cda358d5d184a4383a9716334390022, State: Running, Role: LEADER
I20260807 03:38:13.755686 17496 raft_consensus.cc:3037] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:13.756198 17366 consensus_queue.cc:260] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:13.757064 17366 raft_consensus.cc:3037] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:13.759727 17496 raft_consensus.cc:2454] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c5dca27966a4b8cadb7cc0a8e0d1676 in term 3.
I20260807 03:38:13.760586 17184 leader_election.cc:302] T 80b452de6935470789989a31d5305c47 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2224f937b1b347e2931b7383c84894f2, 6c5dca27966a4b8cadb7cc0a8e0d1676; no voters:
I20260807 03:38:13.761134 17366 raft_consensus.cc:2454] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c5dca27966a4b8cadb7cc0a8e0d1676 in term 3.
I20260807 03:38:13.761176 17763 raft_consensus.cc:2781] T 80b452de6935470789989a31d5305c47 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:38:13.761729 17763 raft_consensus.cc:686] T 80b452de6935470789989a31d5305c47 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 3 LEADER]: Becoming Leader. State: Replica: 6c5dca27966a4b8cadb7cc0a8e0d1676, State: Running, Role: LEADER
I20260807 03:38:13.762480 17763 consensus_queue.cc:237] T 80b452de6935470789989a31d5305c47 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:13.769464 17082 catalog_manager.cc:5495] T 80b452de6935470789989a31d5305c47 P 6c5dca27966a4b8cadb7cc0a8e0d1676 reported cstate change: term changed from 2 to 3, leader changed from 3cda358d5d184a4383a9716334390022 (127.15.198.130) to 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129). New cstate: current_term: 3 leader_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:13.891646 17764 raft_consensus.cc:979] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2: : Instructing follower 6c5dca27966a4b8cadb7cc0a8e0d1676 to start an election
I20260807 03:38:13.892161 17766 raft_consensus.cc:1067] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 5 LEADER]: Signalling peer 6c5dca27966a4b8cadb7cc0a8e0d1676 to start an election
I20260807 03:38:13.893476 17246 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e3f7b02979d648108ef56a2acf13fbf0"
dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
from {username='slave'} at 127.0.0.1:60360
I20260807 03:38:13.893977 17246 raft_consensus.cc:482] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:38:13.894256 17246 raft_consensus.cc:3037] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:38:13.898507 17246 raft_consensus.cc:504] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:13.900516 17246 leader_election.cc:288] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [CANDIDATE]: Term 6 election: Requested vote from peers 3cda358d5d184a4383a9716334390022 (127.15.198.130:45465), 2224f937b1b347e2931b7383c84894f2 (127.15.198.131:41195)
I20260807 03:38:13.900516 17366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f7b02979d648108ef56a2acf13fbf0" candidate_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: true dest_uuid: "3cda358d5d184a4383a9716334390022"
I20260807 03:38:13.901183 17366 raft_consensus.cc:3037] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:38:13.901444 17496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f7b02979d648108ef56a2acf13fbf0" candidate_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: true dest_uuid: "2224f937b1b347e2931b7383c84894f2"
I20260807 03:38:13.901927 17496 raft_consensus.cc:3032] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 03:38:13.902112 17496 raft_consensus.cc:729] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 2224f937b1b347e2931b7383c84894f2, State: Running, Role: LEADER
I20260807 03:38:13.902830 17496 consensus_queue.cc:260] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:13.903859 17496 raft_consensus.cc:3037] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:38:13.906265 17366 raft_consensus.cc:2454] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c5dca27966a4b8cadb7cc0a8e0d1676 in term 6.
I20260807 03:38:13.907158 17184 leader_election.cc:302] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cda358d5d184a4383a9716334390022, 6c5dca27966a4b8cadb7cc0a8e0d1676; no voters:
I20260807 03:38:13.907871 17763 raft_consensus.cc:2781] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 03:38:13.908283 17763 raft_consensus.cc:686] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 6 LEADER]: Becoming Leader. State: Replica: 6c5dca27966a4b8cadb7cc0a8e0d1676, State: Running, Role: LEADER
I20260807 03:38:13.908459 17496 raft_consensus.cc:2454] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c5dca27966a4b8cadb7cc0a8e0d1676 in term 6.
I20260807 03:38:13.909073 17763 consensus_queue.cc:237] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:13.915820 17082 catalog_manager.cc:5495] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 reported cstate change: term changed from 5 to 6, leader changed from 2224f937b1b347e2931b7383c84894f2 (127.15.198.131) to 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129). New cstate: current_term: 6 leader_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:13.955260 17778 raft_consensus.cc:979] T 64fc83d22ad945ad890e67f9763fe413 P 3cda358d5d184a4383a9716334390022: : Instructing follower 6c5dca27966a4b8cadb7cc0a8e0d1676 to start an election
I20260807 03:38:13.955672 17765 raft_consensus.cc:1067] T 64fc83d22ad945ad890e67f9763fe413 P 3cda358d5d184a4383a9716334390022 [term 1 LEADER]: Signalling peer 6c5dca27966a4b8cadb7cc0a8e0d1676 to start an election
I20260807 03:38:13.956930 17246 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "64fc83d22ad945ad890e67f9763fe413"
dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
from {username='slave'} at 127.0.0.1:60348
I20260807 03:38:13.957319 17246 raft_consensus.cc:482] T 64fc83d22ad945ad890e67f9763fe413 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:38:13.957567 17246 raft_consensus.cc:3037] T 64fc83d22ad945ad890e67f9763fe413 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:13.961499 17246 raft_consensus.cc:504] T 64fc83d22ad945ad890e67f9763fe413 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:13.962844 17246 leader_election.cc:288] T 64fc83d22ad945ad890e67f9763fe413 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [CANDIDATE]: Term 2 election: Requested vote from peers 3cda358d5d184a4383a9716334390022 (127.15.198.130:45465), 2224f937b1b347e2931b7383c84894f2 (127.15.198.131:41195)
I20260807 03:38:13.963482 17366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64fc83d22ad945ad890e67f9763fe413" candidate_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3cda358d5d184a4383a9716334390022"
I20260807 03:38:13.964043 17366 raft_consensus.cc:3032] T 64fc83d22ad945ad890e67f9763fe413 P 3cda358d5d184a4383a9716334390022 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:13.963905 17496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64fc83d22ad945ad890e67f9763fe413" candidate_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2224f937b1b347e2931b7383c84894f2"
I20260807 03:38:13.964416 17366 raft_consensus.cc:729] T 64fc83d22ad945ad890e67f9763fe413 P 3cda358d5d184a4383a9716334390022 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3cda358d5d184a4383a9716334390022, State: Running, Role: LEADER
I20260807 03:38:13.964596 17496 raft_consensus.cc:3037] T 64fc83d22ad945ad890e67f9763fe413 P 2224f937b1b347e2931b7383c84894f2 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:13.965235 17366 consensus_queue.cc:260] T 64fc83d22ad945ad890e67f9763fe413 P 3cda358d5d184a4383a9716334390022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:13.966367 17366 raft_consensus.cc:3037] T 64fc83d22ad945ad890e67f9763fe413 P 3cda358d5d184a4383a9716334390022 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:13.969106 17496 raft_consensus.cc:2454] T 64fc83d22ad945ad890e67f9763fe413 P 2224f937b1b347e2931b7383c84894f2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c5dca27966a4b8cadb7cc0a8e0d1676 in term 2.
I20260807 03:38:13.969830 17184 leader_election.cc:302] T 64fc83d22ad945ad890e67f9763fe413 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2224f937b1b347e2931b7383c84894f2, 6c5dca27966a4b8cadb7cc0a8e0d1676; no voters:
I20260807 03:38:13.970331 17366 raft_consensus.cc:2454] T 64fc83d22ad945ad890e67f9763fe413 P 3cda358d5d184a4383a9716334390022 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c5dca27966a4b8cadb7cc0a8e0d1676 in term 2.
I20260807 03:38:13.970435 17763 raft_consensus.cc:2781] T 64fc83d22ad945ad890e67f9763fe413 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:38:13.970950 17763 raft_consensus.cc:686] T 64fc83d22ad945ad890e67f9763fe413 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 LEADER]: Becoming Leader. State: Replica: 6c5dca27966a4b8cadb7cc0a8e0d1676, State: Running, Role: LEADER
I20260807 03:38:13.971673 17763 consensus_queue.cc:237] T 64fc83d22ad945ad890e67f9763fe413 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:13.978569 17082 catalog_manager.cc:5495] T 64fc83d22ad945ad890e67f9763fe413 P 6c5dca27966a4b8cadb7cc0a8e0d1676 reported cstate change: term changed from 1 to 2, leader changed from 3cda358d5d184a4383a9716334390022 (127.15.198.130) to 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129). New cstate: current_term: 2 leader_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:14.132414 17764 raft_consensus.cc:979] T 96451ce78a6f45479178c8d596430538 P 2224f937b1b347e2931b7383c84894f2: : Instructing follower 6c5dca27966a4b8cadb7cc0a8e0d1676 to start an election
I20260807 03:38:14.132992 17766 raft_consensus.cc:1067] T 96451ce78a6f45479178c8d596430538 P 2224f937b1b347e2931b7383c84894f2 [term 3 LEADER]: Signalling peer 6c5dca27966a4b8cadb7cc0a8e0d1676 to start an election
I20260807 03:38:14.133683 17366 raft_consensus.cc:1261] T efcd19aa11284c08ab3b4e1854a17dd9 P 3cda358d5d184a4383a9716334390022 [term 4 FOLLOWER]: Refusing update from remote peer 6c5dca27966a4b8cadb7cc0a8e0d1676: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260807 03:38:14.134526 17246 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "96451ce78a6f45479178c8d596430538"
dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
from {username='slave'} at 127.0.0.1:60360
I20260807 03:38:14.135079 17246 raft_consensus.cc:482] T 96451ce78a6f45479178c8d596430538 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:38:14.134920 17763 consensus_queue.cc:1035] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:38:14.135425 17246 raft_consensus.cc:3037] T 96451ce78a6f45479178c8d596430538 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:38:14.141683 17246 raft_consensus.cc:504] T 96451ce78a6f45479178c8d596430538 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } }
I20260807 03:38:14.143338 17246 leader_election.cc:288] T 96451ce78a6f45479178c8d596430538 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [CANDIDATE]: Term 4 election: Requested vote from peers 2224f937b1b347e2931b7383c84894f2 (127.15.198.131:41195), 3cda358d5d184a4383a9716334390022 (127.15.198.130:45465)
I20260807 03:38:14.145157 17496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96451ce78a6f45479178c8d596430538" candidate_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "2224f937b1b347e2931b7383c84894f2"
I20260807 03:38:14.145504 17367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96451ce78a6f45479178c8d596430538" candidate_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "3cda358d5d184a4383a9716334390022"
I20260807 03:38:14.145727 17496 raft_consensus.cc:3032] T 96451ce78a6f45479178c8d596430538 P 2224f937b1b347e2931b7383c84894f2 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:38:14.146018 17496 raft_consensus.cc:729] T 96451ce78a6f45479178c8d596430538 P 2224f937b1b347e2931b7383c84894f2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2224f937b1b347e2931b7383c84894f2, State: Running, Role: LEADER
I20260807 03:38:14.146847 17496 consensus_queue.cc:260] T 96451ce78a6f45479178c8d596430538 P 2224f937b1b347e2931b7383c84894f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } }
I20260807 03:38:14.148205 17496 raft_consensus.cc:3037] T 96451ce78a6f45479178c8d596430538 P 2224f937b1b347e2931b7383c84894f2 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:38:14.149506 17493 raft_consensus.cc:1261] T efcd19aa11284c08ab3b4e1854a17dd9 P 2224f937b1b347e2931b7383c84894f2 [term 4 FOLLOWER]: Refusing update from remote peer 6c5dca27966a4b8cadb7cc0a8e0d1676: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260807 03:38:14.151177 17768 consensus_queue.cc:1035] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 03:38:14.146056 17367 raft_consensus.cc:3037] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:38:14.164526 17367 raft_consensus.cc:2454] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c5dca27966a4b8cadb7cc0a8e0d1676 in term 4.
I20260807 03:38:14.165004 17496 raft_consensus.cc:2454] T 96451ce78a6f45479178c8d596430538 P 2224f937b1b347e2931b7383c84894f2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c5dca27966a4b8cadb7cc0a8e0d1676 in term 4.
I20260807 03:38:14.165755 17184 leader_election.cc:302] T 96451ce78a6f45479178c8d596430538 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cda358d5d184a4383a9716334390022, 6c5dca27966a4b8cadb7cc0a8e0d1676; no voters:
I20260807 03:38:14.167044 17763 raft_consensus.cc:2781] T 96451ce78a6f45479178c8d596430538 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 03:38:14.167510 17763 raft_consensus.cc:686] T 96451ce78a6f45479178c8d596430538 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 4 LEADER]: Becoming Leader. State: Replica: 6c5dca27966a4b8cadb7cc0a8e0d1676, State: Running, Role: LEADER
I20260807 03:38:14.168895 17763 consensus_queue.cc:237] T 96451ce78a6f45479178c8d596430538 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } }
I20260807 03:38:14.175799 17082 catalog_manager.cc:5495] T 96451ce78a6f45479178c8d596430538 P 6c5dca27966a4b8cadb7cc0a8e0d1676 reported cstate change: term changed from 3 to 4, leader changed from 2224f937b1b347e2931b7383c84894f2 (127.15.198.131) to 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129). New cstate: current_term: 4 leader_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } health_report { overall_health: HEALTHY } } }
I20260807 03:38:14.213366 17496 raft_consensus.cc:1261] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 [term 3 FOLLOWER]: Refusing update from remote peer 6c5dca27966a4b8cadb7cc0a8e0d1676: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 03:38:14.214550 17763 consensus_queue.cc:1035] T 80b452de6935470789989a31d5305c47 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:38:14.222898 17366 raft_consensus.cc:1261] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022 [term 3 FOLLOWER]: Refusing update from remote peer 6c5dca27966a4b8cadb7cc0a8e0d1676: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 03:38:14.224845 17786 consensus_queue.cc:1035] T 80b452de6935470789989a31d5305c47 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 03:38:14.319015 17496 raft_consensus.cc:1261] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 6 FOLLOWER]: Refusing update from remote peer 6c5dca27966a4b8cadb7cc0a8e0d1676: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260807 03:38:14.320253 17786 consensus_queue.cc:1035] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:38:14.328351 17366 raft_consensus.cc:1261] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022 [term 6 FOLLOWER]: Refusing update from remote peer 6c5dca27966a4b8cadb7cc0a8e0d1676: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260807 03:38:14.330300 17786 consensus_queue.cc:1035] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:38:14.379593 17496 raft_consensus.cc:1261] T 64fc83d22ad945ad890e67f9763fe413 P 2224f937b1b347e2931b7383c84894f2 [term 2 FOLLOWER]: Refusing update from remote peer 6c5dca27966a4b8cadb7cc0a8e0d1676: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:38:14.380676 17786 consensus_queue.cc:1035] T 64fc83d22ad945ad890e67f9763fe413 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:14.388530 17366 raft_consensus.cc:1261] T 64fc83d22ad945ad890e67f9763fe413 P 3cda358d5d184a4383a9716334390022 [term 2 FOLLOWER]: Refusing update from remote peer 6c5dca27966a4b8cadb7cc0a8e0d1676: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:38:14.389943 17786 consensus_queue.cc:1035] T 64fc83d22ad945ad890e67f9763fe413 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:14.638756 17366 raft_consensus.cc:1261] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022 [term 4 FOLLOWER]: Refusing update from remote peer 6c5dca27966a4b8cadb7cc0a8e0d1676: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260807 03:38:14.640066 17786 consensus_queue.cc:1035] T 96451ce78a6f45479178c8d596430538 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:38:14.651152 17496 raft_consensus.cc:1261] T 96451ce78a6f45479178c8d596430538 P 2224f937b1b347e2931b7383c84894f2 [term 4 FOLLOWER]: Refusing update from remote peer 6c5dca27966a4b8cadb7cc0a8e0d1676: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260807 03:38:14.652511 17763 consensus_queue.cc:1035] T 96451ce78a6f45479178c8d596430538 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:38:16.658692 16154 auto_leader_rebalancer-test.cc:373] The leader distribution is
I20260807 03:38:16.659478 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:38:16.666994 17246 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "80b452de6935470789989a31d5305c47"
dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
mode: GRACEFUL
new_leader_uuid: "2224f937b1b347e2931b7383c84894f2"
from {username='slave'} at 127.0.0.1:60390
I20260807 03:38:16.667686 17246 raft_consensus.cc:595] T 80b452de6935470789989a31d5305c47 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 3 LEADER]: Received request to transfer leadership to TS 2224f937b1b347e2931b7383c84894f2
I20260807 03:38:16.670073 17246 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8f6f16d016d3437d8aeae4a64c42a856"
dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
mode: GRACEFUL
new_leader_uuid: "3cda358d5d184a4383a9716334390022"
from {username='slave'} at 127.0.0.1:60390
I20260807 03:38:16.670547 17246 raft_consensus.cc:595] T 8f6f16d016d3437d8aeae4a64c42a856 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 LEADER]: Received request to transfer leadership to TS 3cda358d5d184a4383a9716334390022
I20260807 03:38:16.672713 17246 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a41a7b13d3284dcd8b9b616eeb93c578"
dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
mode: GRACEFUL
new_leader_uuid: "3cda358d5d184a4383a9716334390022"
from {username='slave'} at 127.0.0.1:60390
I20260807 03:38:16.673213 17246 raft_consensus.cc:595] T a41a7b13d3284dcd8b9b616eeb93c578 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 LEADER]: Received request to transfer leadership to TS 3cda358d5d184a4383a9716334390022
I20260807 03:38:16.675304 17246 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f99d56107b6a48db9611f1ce8282d575"
dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
mode: GRACEFUL
new_leader_uuid: "2224f937b1b347e2931b7383c84894f2"
from {username='slave'} at 127.0.0.1:60390
I20260807 03:38:16.675733 17246 raft_consensus.cc:595] T f99d56107b6a48db9611f1ce8282d575 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 LEADER]: Received request to transfer leadership to TS 2224f937b1b347e2931b7383c84894f2
I20260807 03:38:16.676610 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260807 03:38:16.677008 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:38:16.750659 17808 raft_consensus.cc:979] T a41a7b13d3284dcd8b9b616eeb93c578 P 6c5dca27966a4b8cadb7cc0a8e0d1676: : Instructing follower 3cda358d5d184a4383a9716334390022 to start an election
I20260807 03:38:16.751236 17812 raft_consensus.cc:1067] T a41a7b13d3284dcd8b9b616eeb93c578 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 LEADER]: Signalling peer 3cda358d5d184a4383a9716334390022 to start an election
I20260807 03:38:16.752673 17367 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a41a7b13d3284dcd8b9b616eeb93c578"
dest_uuid: "3cda358d5d184a4383a9716334390022"
from {username='slave'} at 127.0.0.1:52434
I20260807 03:38:16.753191 17367 raft_consensus.cc:482] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:38:16.753484 17367 raft_consensus.cc:3037] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:16.760169 17367 raft_consensus.cc:504] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:16.762082 17367 leader_election.cc:288] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022 [CANDIDATE]: Term 3 election: Requested vote from peers 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887), 2224f937b1b347e2931b7383c84894f2 (127.15.198.131:41195)
I20260807 03:38:16.762832 17246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a41a7b13d3284dcd8b9b616eeb93c578" candidate_uuid: "3cda358d5d184a4383a9716334390022" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
I20260807 03:38:16.763191 17496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a41a7b13d3284dcd8b9b616eeb93c578" candidate_uuid: "3cda358d5d184a4383a9716334390022" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2224f937b1b347e2931b7383c84894f2"
I20260807 03:38:16.763499 17246 raft_consensus.cc:3032] T a41a7b13d3284dcd8b9b616eeb93c578 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 03:38:16.763696 17496 raft_consensus.cc:3037] T a41a7b13d3284dcd8b9b616eeb93c578 P 2224f937b1b347e2931b7383c84894f2 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:16.763841 17246 raft_consensus.cc:729] T a41a7b13d3284dcd8b9b616eeb93c578 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6c5dca27966a4b8cadb7cc0a8e0d1676, State: Running, Role: LEADER
I20260807 03:38:16.764667 17246 consensus_queue.cc:260] T a41a7b13d3284dcd8b9b616eeb93c578 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:16.765657 17246 raft_consensus.cc:3037] T a41a7b13d3284dcd8b9b616eeb93c578 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:16.768803 17496 raft_consensus.cc:2454] T a41a7b13d3284dcd8b9b616eeb93c578 P 2224f937b1b347e2931b7383c84894f2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cda358d5d184a4383a9716334390022 in term 3.
I20260807 03:38:16.769856 17316 leader_election.cc:302] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2224f937b1b347e2931b7383c84894f2, 3cda358d5d184a4383a9716334390022; no voters:
I20260807 03:38:16.770676 17765 raft_consensus.cc:2781] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:38:16.770691 17246 raft_consensus.cc:2454] T a41a7b13d3284dcd8b9b616eeb93c578 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cda358d5d184a4383a9716334390022 in term 3.
I20260807 03:38:16.771716 17765 raft_consensus.cc:686] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022 [term 3 LEADER]: Becoming Leader. State: Replica: 3cda358d5d184a4383a9716334390022, State: Running, Role: LEADER
I20260807 03:38:16.772619 17765 consensus_queue.cc:237] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:16.778792 17082 catalog_manager.cc:5495] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022 reported cstate change: term changed from 2 to 3, leader changed from 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129) to 3cda358d5d184a4383a9716334390022 (127.15.198.130). New cstate: current_term: 3 leader_uuid: "3cda358d5d184a4383a9716334390022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:16.832803 17812 raft_consensus.cc:979] T 80b452de6935470789989a31d5305c47 P 6c5dca27966a4b8cadb7cc0a8e0d1676: : Instructing follower 2224f937b1b347e2931b7383c84894f2 to start an election
I20260807 03:38:16.833348 17808 raft_consensus.cc:1067] T 80b452de6935470789989a31d5305c47 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 3 LEADER]: Signalling peer 2224f937b1b347e2931b7383c84894f2 to start an election
I20260807 03:38:16.834785 17496 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "80b452de6935470789989a31d5305c47"
dest_uuid: "2224f937b1b347e2931b7383c84894f2"
from {username='slave'} at 127.0.0.1:48312
I20260807 03:38:16.835268 17496 raft_consensus.cc:482] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:38:16.835595 17496 raft_consensus.cc:3037] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:38:16.842422 17496 raft_consensus.cc:504] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:16.845490 17246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b452de6935470789989a31d5305c47" candidate_uuid: "2224f937b1b347e2931b7383c84894f2" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
I20260807 03:38:16.845754 17496 leader_election.cc:288] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 [CANDIDATE]: Term 4 election: Requested vote from peers 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887), 3cda358d5d184a4383a9716334390022 (127.15.198.130:45465)
I20260807 03:38:16.845729 17367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b452de6935470789989a31d5305c47" candidate_uuid: "2224f937b1b347e2931b7383c84894f2" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "3cda358d5d184a4383a9716334390022"
I20260807 03:38:16.846057 17246 raft_consensus.cc:3032] T 80b452de6935470789989a31d5305c47 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:38:16.846352 17246 raft_consensus.cc:729] T 80b452de6935470789989a31d5305c47 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6c5dca27966a4b8cadb7cc0a8e0d1676, State: Running, Role: LEADER
I20260807 03:38:16.846436 17367 raft_consensus.cc:3037] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:38:16.847106 17246 consensus_queue.cc:260] T 80b452de6935470789989a31d5305c47 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:16.848129 17246 raft_consensus.cc:3037] T 80b452de6935470789989a31d5305c47 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:38:16.852587 17367 raft_consensus.cc:2454] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2224f937b1b347e2931b7383c84894f2 in term 4.
I20260807 03:38:16.853817 17443 leader_election.cc:302] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2224f937b1b347e2931b7383c84894f2, 3cda358d5d184a4383a9716334390022; no voters:
I20260807 03:38:16.854475 17246 raft_consensus.cc:2454] T 80b452de6935470789989a31d5305c47 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2224f937b1b347e2931b7383c84894f2 in term 4.
I20260807 03:38:16.854826 17766 raft_consensus.cc:2781] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 03:38:16.856343 17766 raft_consensus.cc:686] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 [term 4 LEADER]: Becoming Leader. State: Replica: 2224f937b1b347e2931b7383c84894f2, State: Running, Role: LEADER
I20260807 03:38:16.857355 17766 consensus_queue.cc:237] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:16.866714 17082 catalog_manager.cc:5495] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 reported cstate change: term changed from 3 to 4, leader changed from 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129) to 2224f937b1b347e2931b7383c84894f2 (127.15.198.131). New cstate: current_term: 4 leader_uuid: "2224f937b1b347e2931b7383c84894f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } health_report { overall_health: HEALTHY } } }
I20260807 03:38:16.900563 17812 raft_consensus.cc:979] T 8f6f16d016d3437d8aeae4a64c42a856 P 6c5dca27966a4b8cadb7cc0a8e0d1676: : Instructing follower 3cda358d5d184a4383a9716334390022 to start an election
I20260807 03:38:16.901098 17808 raft_consensus.cc:1067] T 8f6f16d016d3437d8aeae4a64c42a856 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 LEADER]: Signalling peer 3cda358d5d184a4383a9716334390022 to start an election
I20260807 03:38:16.902467 17367 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8f6f16d016d3437d8aeae4a64c42a856"
dest_uuid: "3cda358d5d184a4383a9716334390022"
from {username='slave'} at 127.0.0.1:52434
I20260807 03:38:16.902985 17367 raft_consensus.cc:482] T 8f6f16d016d3437d8aeae4a64c42a856 P 3cda358d5d184a4383a9716334390022 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:38:16.903259 17367 raft_consensus.cc:3037] T 8f6f16d016d3437d8aeae4a64c42a856 P 3cda358d5d184a4383a9716334390022 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:16.907435 17367 raft_consensus.cc:504] T 8f6f16d016d3437d8aeae4a64c42a856 P 3cda358d5d184a4383a9716334390022 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:16.908744 17367 leader_election.cc:288] T 8f6f16d016d3437d8aeae4a64c42a856 P 3cda358d5d184a4383a9716334390022 [CANDIDATE]: Term 3 election: Requested vote from peers 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887), 2224f937b1b347e2931b7383c84894f2 (127.15.198.131:41195)
I20260807 03:38:16.909497 17246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f6f16d016d3437d8aeae4a64c42a856" candidate_uuid: "3cda358d5d184a4383a9716334390022" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
I20260807 03:38:16.909636 17496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f6f16d016d3437d8aeae4a64c42a856" candidate_uuid: "3cda358d5d184a4383a9716334390022" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2224f937b1b347e2931b7383c84894f2"
I20260807 03:38:16.910108 17246 raft_consensus.cc:3032] T 8f6f16d016d3437d8aeae4a64c42a856 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 03:38:16.910151 17496 raft_consensus.cc:3037] T 8f6f16d016d3437d8aeae4a64c42a856 P 2224f937b1b347e2931b7383c84894f2 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:16.910421 17246 raft_consensus.cc:729] T 8f6f16d016d3437d8aeae4a64c42a856 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6c5dca27966a4b8cadb7cc0a8e0d1676, State: Running, Role: LEADER
I20260807 03:38:16.911116 17246 consensus_queue.cc:260] T 8f6f16d016d3437d8aeae4a64c42a856 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:16.912020 17246 raft_consensus.cc:3037] T 8f6f16d016d3437d8aeae4a64c42a856 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:16.914909 17496 raft_consensus.cc:2454] T 8f6f16d016d3437d8aeae4a64c42a856 P 2224f937b1b347e2931b7383c84894f2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cda358d5d184a4383a9716334390022 in term 3.
I20260807 03:38:16.915747 17316 leader_election.cc:302] T 8f6f16d016d3437d8aeae4a64c42a856 P 3cda358d5d184a4383a9716334390022 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2224f937b1b347e2931b7383c84894f2, 3cda358d5d184a4383a9716334390022; no voters:
I20260807 03:38:16.916457 17765 raft_consensus.cc:2781] T 8f6f16d016d3437d8aeae4a64c42a856 P 3cda358d5d184a4383a9716334390022 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:38:16.916977 17765 raft_consensus.cc:686] T 8f6f16d016d3437d8aeae4a64c42a856 P 3cda358d5d184a4383a9716334390022 [term 3 LEADER]: Becoming Leader. State: Replica: 3cda358d5d184a4383a9716334390022, State: Running, Role: LEADER
I20260807 03:38:16.917052 17246 raft_consensus.cc:2454] T 8f6f16d016d3437d8aeae4a64c42a856 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cda358d5d184a4383a9716334390022 in term 3.
I20260807 03:38:16.918108 17765 consensus_queue.cc:237] T 8f6f16d016d3437d8aeae4a64c42a856 P 3cda358d5d184a4383a9716334390022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:16.925487 17082 catalog_manager.cc:5495] T 8f6f16d016d3437d8aeae4a64c42a856 P 3cda358d5d184a4383a9716334390022 reported cstate change: term changed from 2 to 3, leader changed from 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129) to 3cda358d5d184a4383a9716334390022 (127.15.198.130). New cstate: current_term: 3 leader_uuid: "3cda358d5d184a4383a9716334390022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:17.001081 17812 raft_consensus.cc:979] T f99d56107b6a48db9611f1ce8282d575 P 6c5dca27966a4b8cadb7cc0a8e0d1676: : Instructing follower 2224f937b1b347e2931b7383c84894f2 to start an election
I20260807 03:38:17.001564 17808 raft_consensus.cc:1067] T f99d56107b6a48db9611f1ce8282d575 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 LEADER]: Signalling peer 2224f937b1b347e2931b7383c84894f2 to start an election
I20260807 03:38:17.002960 17496 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f99d56107b6a48db9611f1ce8282d575"
dest_uuid: "2224f937b1b347e2931b7383c84894f2"
from {username='slave'} at 127.0.0.1:48312
I20260807 03:38:17.003475 17496 raft_consensus.cc:482] T f99d56107b6a48db9611f1ce8282d575 P 2224f937b1b347e2931b7383c84894f2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:38:17.003767 17496 raft_consensus.cc:3037] T f99d56107b6a48db9611f1ce8282d575 P 2224f937b1b347e2931b7383c84894f2 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:17.007995 17496 raft_consensus.cc:504] T f99d56107b6a48db9611f1ce8282d575 P 2224f937b1b347e2931b7383c84894f2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:17.009400 17496 leader_election.cc:288] T f99d56107b6a48db9611f1ce8282d575 P 2224f937b1b347e2931b7383c84894f2 [CANDIDATE]: Term 3 election: Requested vote from peers 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887), 3cda358d5d184a4383a9716334390022 (127.15.198.130:45465)
I20260807 03:38:17.010213 17246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f99d56107b6a48db9611f1ce8282d575" candidate_uuid: "2224f937b1b347e2931b7383c84894f2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676"
I20260807 03:38:17.010425 17367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f99d56107b6a48db9611f1ce8282d575" candidate_uuid: "2224f937b1b347e2931b7383c84894f2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3cda358d5d184a4383a9716334390022"
I20260807 03:38:17.010784 17246 raft_consensus.cc:3032] T f99d56107b6a48db9611f1ce8282d575 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 03:38:17.010887 17367 raft_consensus.cc:3037] T f99d56107b6a48db9611f1ce8282d575 P 3cda358d5d184a4383a9716334390022 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:17.011101 17246 raft_consensus.cc:729] T f99d56107b6a48db9611f1ce8282d575 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6c5dca27966a4b8cadb7cc0a8e0d1676, State: Running, Role: LEADER
I20260807 03:38:17.011886 17246 consensus_queue.cc:260] T f99d56107b6a48db9611f1ce8282d575 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:17.013111 17246 raft_consensus.cc:3037] T f99d56107b6a48db9611f1ce8282d575 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:17.016862 17367 raft_consensus.cc:2454] T f99d56107b6a48db9611f1ce8282d575 P 3cda358d5d184a4383a9716334390022 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2224f937b1b347e2931b7383c84894f2 in term 3.
I20260807 03:38:17.017738 17443 leader_election.cc:302] T f99d56107b6a48db9611f1ce8282d575 P 2224f937b1b347e2931b7383c84894f2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2224f937b1b347e2931b7383c84894f2, 3cda358d5d184a4383a9716334390022; no voters:
I20260807 03:38:17.018420 17766 raft_consensus.cc:2781] T f99d56107b6a48db9611f1ce8282d575 P 2224f937b1b347e2931b7383c84894f2 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:38:17.018873 17766 raft_consensus.cc:686] T f99d56107b6a48db9611f1ce8282d575 P 2224f937b1b347e2931b7383c84894f2 [term 3 LEADER]: Becoming Leader. State: Replica: 2224f937b1b347e2931b7383c84894f2, State: Running, Role: LEADER
I20260807 03:38:17.018885 17246 raft_consensus.cc:2454] T f99d56107b6a48db9611f1ce8282d575 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2224f937b1b347e2931b7383c84894f2 in term 3.
I20260807 03:38:17.019881 17766 consensus_queue.cc:237] T f99d56107b6a48db9611f1ce8282d575 P 2224f937b1b347e2931b7383c84894f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } }
I20260807 03:38:17.026903 17082 catalog_manager.cc:5495] T f99d56107b6a48db9611f1ce8282d575 P 2224f937b1b347e2931b7383c84894f2 reported cstate change: term changed from 2 to 3, leader changed from 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129) to 2224f937b1b347e2931b7383c84894f2 (127.15.198.131). New cstate: current_term: 3 leader_uuid: "2224f937b1b347e2931b7383c84894f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 } health_report { overall_health: HEALTHY } } }
I20260807 03:38:17.156764 17246 raft_consensus.cc:1261] T a41a7b13d3284dcd8b9b616eeb93c578 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 3 FOLLOWER]: Refusing update from remote peer 3cda358d5d184a4383a9716334390022: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 03:38:17.157846 17765 consensus_queue.cc:1035] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:38:17.169117 17496 raft_consensus.cc:1261] T a41a7b13d3284dcd8b9b616eeb93c578 P 2224f937b1b347e2931b7383c84894f2 [term 3 FOLLOWER]: Refusing update from remote peer 3cda358d5d184a4383a9716334390022: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 03:38:17.171113 17829 consensus_queue.cc:1035] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 03:38:17.318306 17367 raft_consensus.cc:1261] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022 [term 4 FOLLOWER]: Refusing update from remote peer 2224f937b1b347e2931b7383c84894f2: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260807 03:38:17.319608 17766 consensus_queue.cc:1035] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:38:17.329047 17246 raft_consensus.cc:1261] T 80b452de6935470789989a31d5305c47 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 4 FOLLOWER]: Refusing update from remote peer 2224f937b1b347e2931b7383c84894f2: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260807 03:38:17.330551 17837 consensus_queue.cc:1035] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 03:38:17.389262 17496 raft_consensus.cc:1261] T 8f6f16d016d3437d8aeae4a64c42a856 P 2224f937b1b347e2931b7383c84894f2 [term 3 FOLLOWER]: Refusing update from remote peer 3cda358d5d184a4383a9716334390022: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 03:38:17.390317 17829 consensus_queue.cc:1035] T 8f6f16d016d3437d8aeae4a64c42a856 P 3cda358d5d184a4383a9716334390022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2224f937b1b347e2931b7383c84894f2" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 41195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:38:17.399420 17246 raft_consensus.cc:1261] T 8f6f16d016d3437d8aeae4a64c42a856 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 3 FOLLOWER]: Refusing update from remote peer 3cda358d5d184a4383a9716334390022: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 03:38:17.401510 17842 consensus_queue.cc:1035] T 8f6f16d016d3437d8aeae4a64c42a856 P 3cda358d5d184a4383a9716334390022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 03:38:17.525954 17367 raft_consensus.cc:1261] T f99d56107b6a48db9611f1ce8282d575 P 3cda358d5d184a4383a9716334390022 [term 3 FOLLOWER]: Refusing update from remote peer 2224f937b1b347e2931b7383c84894f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 03:38:17.527104 17837 consensus_queue.cc:1035] T f99d56107b6a48db9611f1ce8282d575 P 2224f937b1b347e2931b7383c84894f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cda358d5d184a4383a9716334390022" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 45465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:38:17.535763 17246 raft_consensus.cc:1261] T f99d56107b6a48db9611f1ce8282d575 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 3 FOLLOWER]: Refusing update from remote peer 2224f937b1b347e2931b7383c84894f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 03:38:17.537277 17766 consensus_queue.cc:1035] T f99d56107b6a48db9611f1ce8282d575 P 2224f937b1b347e2931b7383c84894f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c5dca27966a4b8cadb7cc0a8e0d1676" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 34887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:38:17.677791 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:38:17.682937 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 03:38:17.683328 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:38:18.684145 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:38:18.689122 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 03:38:18.689525 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:38:19.690234 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:38:19.695556 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 03:38:19.696003 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:38:20.696790 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:38:20.701918 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 03:38:20.702221 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:38:21.703112 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:38:21.708344 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 03:38:21.708738 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:38:22.709450 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:38:22.716188 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 03:38:22.716632 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:38:23.717516 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:38:23.723143 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 03:38:23.723527 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:38:24.724270 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:38:24.730034 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 03:38:24.730458 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:38:25.731279 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:38:25.736351 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 03:38:25.736747 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:38:26.765924 16154 auto_leader_rebalancer-test.cc:384] The leader distribution is
I20260807 03:38:26.766572 16154 tablet_server.cc:178] TabletServer@127.15.198.129:0 shutting down...
I20260807 03:38:26.798630 16154 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260807 03:38:26.799484 16154 tablet_replica.cc:330] T c93d1700ec3248b3a169ff40a4186010 P 6c5dca27966a4b8cadb7cc0a8e0d1676: stopping tablet replica
W20260807 03:38:26.799696 17440 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 3 similar messages]
I20260807 03:38:26.800799 16154 raft_consensus.cc:2229] T c93d1700ec3248b3a169ff40a4186010 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:38:26.801766 16154 raft_consensus.cc:2258] T c93d1700ec3248b3a169ff40a4186010 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260807 03:38:26.803099 17440 consensus_peers.cc:487] T c93d1700ec3248b3a169ff40a4186010 P 2224f937b1b347e2931b7383c84894f2 -> Peer 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887): Couldn't send request to peer 6c5dca27966a4b8cadb7cc0a8e0d1676. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260807 03:38:26.804557 16154 tablet_replica.cc:330] T 64fc83d22ad945ad890e67f9763fe413 P 6c5dca27966a4b8cadb7cc0a8e0d1676: stopping tablet replica
I20260807 03:38:26.805079 16154 raft_consensus.cc:2229] T 64fc83d22ad945ad890e67f9763fe413 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 LEADER]: Raft consensus shutting down.
I20260807 03:38:26.805927 16154 raft_consensus.cc:2258] T 64fc83d22ad945ad890e67f9763fe413 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260807 03:38:26.807662 16154 tablet_replica.cc:330] T 96451ce78a6f45479178c8d596430538 P 6c5dca27966a4b8cadb7cc0a8e0d1676: stopping tablet replica
I20260807 03:38:26.808257 16154 raft_consensus.cc:2229] T 96451ce78a6f45479178c8d596430538 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 4 LEADER]: Raft consensus shutting down.
I20260807 03:38:26.809041 16154 raft_consensus.cc:2258] T 96451ce78a6f45479178c8d596430538 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260807 03:38:26.810570 16154 tablet_replica.cc:330] T f99d56107b6a48db9611f1ce8282d575 P 6c5dca27966a4b8cadb7cc0a8e0d1676: stopping tablet replica
I20260807 03:38:26.811035 16154 raft_consensus.cc:2229] T f99d56107b6a48db9611f1ce8282d575 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260807 03:38:26.811407 16154 raft_consensus.cc:2258] T f99d56107b6a48db9611f1ce8282d575 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 03:38:26.813426 16154 tablet_replica.cc:330] T 80b452de6935470789989a31d5305c47 P 6c5dca27966a4b8cadb7cc0a8e0d1676: stopping tablet replica
I20260807 03:38:26.813951 16154 raft_consensus.cc:2229] T 80b452de6935470789989a31d5305c47 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260807 03:38:26.814424 16154 raft_consensus.cc:2258] T 80b452de6935470789989a31d5305c47 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260807 03:38:26.816368 16154 tablet_replica.cc:330] T 6f881ad9daec4be7b7ef3b0475c45474 P 6c5dca27966a4b8cadb7cc0a8e0d1676: stopping tablet replica
I20260807 03:38:26.817140 16154 raft_consensus.cc:2229] T 6f881ad9daec4be7b7ef3b0475c45474 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:38:26.817559 16154 raft_consensus.cc:2258] T 6f881ad9daec4be7b7ef3b0475c45474 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:38:26.819636 16154 tablet_replica.cc:330] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676: stopping tablet replica
I20260807 03:38:26.820065 16154 raft_consensus.cc:2229] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 4 LEADER]: Raft consensus shutting down.
I20260807 03:38:26.820894 16154 raft_consensus.cc:2258] T efcd19aa11284c08ab3b4e1854a17dd9 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260807 03:38:26.822845 16154 tablet_replica.cc:330] T a41a7b13d3284dcd8b9b616eeb93c578 P 6c5dca27966a4b8cadb7cc0a8e0d1676: stopping tablet replica
I20260807 03:38:26.823308 16154 raft_consensus.cc:2229] T a41a7b13d3284dcd8b9b616eeb93c578 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260807 03:38:26.823712 16154 raft_consensus.cc:2258] T a41a7b13d3284dcd8b9b616eeb93c578 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260807 03:38:26.825124 17440 consensus_peers.cc:487] T f99d56107b6a48db9611f1ce8282d575 P 2224f937b1b347e2931b7383c84894f2 -> Peer 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887): Couldn't send request to peer 6c5dca27966a4b8cadb7cc0a8e0d1676. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260807 03:38:26.825663 16154 tablet_replica.cc:330] T 8f6f16d016d3437d8aeae4a64c42a856 P 6c5dca27966a4b8cadb7cc0a8e0d1676: stopping tablet replica
I20260807 03:38:26.826186 16154 raft_consensus.cc:2229] T 8f6f16d016d3437d8aeae4a64c42a856 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260807 03:38:26.826599 16154 raft_consensus.cc:2258] T 8f6f16d016d3437d8aeae4a64c42a856 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 03:38:26.828318 16154 tablet_replica.cc:330] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676: stopping tablet replica
I20260807 03:38:26.828751 16154 raft_consensus.cc:2229] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 6 LEADER]: Raft consensus shutting down.
I20260807 03:38:26.829577 16154 raft_consensus.cc:2258] T e3f7b02979d648108ef56a2acf13fbf0 P 6c5dca27966a4b8cadb7cc0a8e0d1676 [term 6 FOLLOWER]: Raft consensus is shut down!
I20260807 03:38:26.851220 16154 tablet_server.cc:194] TabletServer@127.15.198.129:0 shutdown complete.
W20260807 03:38:26.852952 17313 consensus_peers.cc:487] T 6f881ad9daec4be7b7ef3b0475c45474 P 3cda358d5d184a4383a9716334390022 -> Peer 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887): Couldn't send request to peer 6c5dca27966a4b8cadb7cc0a8e0d1676. Status: Network error: Client connection negotiation failed: client connection to 127.15.198.129:34887: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260807 03:38:26.853421 17313 consensus_peers.cc:487] T 8f6f16d016d3437d8aeae4a64c42a856 P 3cda358d5d184a4383a9716334390022 -> Peer 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887): Couldn't send request to peer 6c5dca27966a4b8cadb7cc0a8e0d1676. Status: Network error: Client connection negotiation failed: client connection to 127.15.198.129:34887: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 03:38:26.870220 16154 tablet_server.cc:178] TabletServer@127.15.198.130:0 shutting down...
I20260807 03:38:26.900732 16154 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260807 03:38:26.901500 16154 tablet_replica.cc:330] T c93d1700ec3248b3a169ff40a4186010 P 3cda358d5d184a4383a9716334390022: stopping tablet replica
I20260807 03:38:26.902163 16154 raft_consensus.cc:2229] T c93d1700ec3248b3a169ff40a4186010 P 3cda358d5d184a4383a9716334390022 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:38:26.902627 16154 raft_consensus.cc:2258] T c93d1700ec3248b3a169ff40a4186010 P 3cda358d5d184a4383a9716334390022 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:38:26.904492 16154 tablet_replica.cc:330] T 64fc83d22ad945ad890e67f9763fe413 P 3cda358d5d184a4383a9716334390022: stopping tablet replica
I20260807 03:38:26.904994 16154 raft_consensus.cc:2229] T 64fc83d22ad945ad890e67f9763fe413 P 3cda358d5d184a4383a9716334390022 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260807 03:38:26.905421 16154 raft_consensus.cc:2258] T 64fc83d22ad945ad890e67f9763fe413 P 3cda358d5d184a4383a9716334390022 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260807 03:38:26.907002 16154 tablet_replica.cc:330] T efcd19aa11284c08ab3b4e1854a17dd9 P 3cda358d5d184a4383a9716334390022: stopping tablet replica
I20260807 03:38:26.907475 16154 raft_consensus.cc:2229] T efcd19aa11284c08ab3b4e1854a17dd9 P 3cda358d5d184a4383a9716334390022 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260807 03:38:26.907892 16154 raft_consensus.cc:2258] T efcd19aa11284c08ab3b4e1854a17dd9 P 3cda358d5d184a4383a9716334390022 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260807 03:38:26.909363 16154 tablet_replica.cc:330] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022: stopping tablet replica
I20260807 03:38:26.909830 16154 raft_consensus.cc:2229] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260807 03:38:26.910269 16154 raft_consensus.cc:2258] T 80b452de6935470789989a31d5305c47 P 3cda358d5d184a4383a9716334390022 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260807 03:38:26.911789 16154 tablet_replica.cc:330] T 6f881ad9daec4be7b7ef3b0475c45474 P 3cda358d5d184a4383a9716334390022: stopping tablet replica
I20260807 03:38:26.912317 16154 raft_consensus.cc:2229] T 6f881ad9daec4be7b7ef3b0475c45474 P 3cda358d5d184a4383a9716334390022 [term 1 LEADER]: Raft consensus shutting down.
I20260807 03:38:26.913172 16154 raft_consensus.cc:2258] T 6f881ad9daec4be7b7ef3b0475c45474 P 3cda358d5d184a4383a9716334390022 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:38:26.914819 16154 tablet_replica.cc:330] T 8f6f16d016d3437d8aeae4a64c42a856 P 3cda358d5d184a4383a9716334390022: stopping tablet replica
I20260807 03:38:26.915279 16154 raft_consensus.cc:2229] T 8f6f16d016d3437d8aeae4a64c42a856 P 3cda358d5d184a4383a9716334390022 [term 3 LEADER]: Raft consensus shutting down.
I20260807 03:38:26.916056 16154 raft_consensus.cc:2258] T 8f6f16d016d3437d8aeae4a64c42a856 P 3cda358d5d184a4383a9716334390022 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 03:38:26.917857 16154 tablet_replica.cc:330] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022: stopping tablet replica
I20260807 03:38:26.918367 16154 raft_consensus.cc:2229] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022 [term 3 LEADER]: Raft consensus shutting down.
I20260807 03:38:26.919178 16154 raft_consensus.cc:2258] T a41a7b13d3284dcd8b9b616eeb93c578 P 3cda358d5d184a4383a9716334390022 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 03:38:26.920958 16154 tablet_replica.cc:330] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022: stopping tablet replica
I20260807 03:38:26.921404 16154 raft_consensus.cc:2229] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022 [term 6 FOLLOWER]: Raft consensus shutting down.
I20260807 03:38:26.921903 16154 raft_consensus.cc:2258] T e3f7b02979d648108ef56a2acf13fbf0 P 3cda358d5d184a4383a9716334390022 [term 6 FOLLOWER]: Raft consensus is shut down!
I20260807 03:38:26.923446 16154 tablet_replica.cc:330] T f99d56107b6a48db9611f1ce8282d575 P 3cda358d5d184a4383a9716334390022: stopping tablet replica
I20260807 03:38:26.923870 16154 raft_consensus.cc:2229] T f99d56107b6a48db9611f1ce8282d575 P 3cda358d5d184a4383a9716334390022 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260807 03:38:26.924352 16154 raft_consensus.cc:2258] T f99d56107b6a48db9611f1ce8282d575 P 3cda358d5d184a4383a9716334390022 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 03:38:26.925829 16154 tablet_replica.cc:330] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022: stopping tablet replica
I20260807 03:38:26.926282 16154 raft_consensus.cc:2229] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260807 03:38:26.926728 16154 raft_consensus.cc:2258] T 96451ce78a6f45479178c8d596430538 P 3cda358d5d184a4383a9716334390022 [term 4 FOLLOWER]: Raft consensus is shut down!
W20260807 03:38:26.933943 17440 consensus_peers.cc:487] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 -> Peer 6c5dca27966a4b8cadb7cc0a8e0d1676 (127.15.198.129:34887): Couldn't send request to peer 6c5dca27966a4b8cadb7cc0a8e0d1676. Status: Network error: Client connection negotiation failed: client connection to 127.15.198.129:34887: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 03:38:26.957129 16154 tablet_server.cc:194] TabletServer@127.15.198.130:0 shutdown complete.
I20260807 03:38:26.974823 16154 tablet_server.cc:178] TabletServer@127.15.198.131:0 shutting down...
I20260807 03:38:27.006417 16154 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260807 03:38:27.007037 16154 tablet_replica.cc:330] T 6f881ad9daec4be7b7ef3b0475c45474 P 2224f937b1b347e2931b7383c84894f2: stopping tablet replica
I20260807 03:38:27.007745 16154 raft_consensus.cc:2229] T 6f881ad9daec4be7b7ef3b0475c45474 P 2224f937b1b347e2931b7383c84894f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:38:27.008276 16154 raft_consensus.cc:2258] T 6f881ad9daec4be7b7ef3b0475c45474 P 2224f937b1b347e2931b7383c84894f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:38:27.010138 16154 tablet_replica.cc:330] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2: stopping tablet replica
I20260807 03:38:27.010710 16154 raft_consensus.cc:2229] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 [term 4 LEADER]: Raft consensus shutting down.
I20260807 03:38:27.011535 16154 raft_consensus.cc:2258] T 80b452de6935470789989a31d5305c47 P 2224f937b1b347e2931b7383c84894f2 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260807 03:38:27.013368 16154 tablet_replica.cc:330] T efcd19aa11284c08ab3b4e1854a17dd9 P 2224f937b1b347e2931b7383c84894f2: stopping tablet replica
I20260807 03:38:27.013897 16154 raft_consensus.cc:2229] T efcd19aa11284c08ab3b4e1854a17dd9 P 2224f937b1b347e2931b7383c84894f2 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260807 03:38:27.014297 16154 raft_consensus.cc:2258] T efcd19aa11284c08ab3b4e1854a17dd9 P 2224f937b1b347e2931b7383c84894f2 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260807 03:38:27.015846 16154 tablet_replica.cc:330] T c93d1700ec3248b3a169ff40a4186010 P 2224f937b1b347e2931b7383c84894f2: stopping tablet replica
I20260807 03:38:27.016446 16154 raft_consensus.cc:2229] T c93d1700ec3248b3a169ff40a4186010 P 2224f937b1b347e2931b7383c84894f2 [term 1 LEADER]: Raft consensus shutting down.
I20260807 03:38:27.017237 16154 raft_consensus.cc:2258] T c93d1700ec3248b3a169ff40a4186010 P 2224f937b1b347e2931b7383c84894f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:38:27.019147 16154 tablet_replica.cc:330] T 96451ce78a6f45479178c8d596430538 P 2224f937b1b347e2931b7383c84894f2: stopping tablet replica
I20260807 03:38:27.019623 16154 raft_consensus.cc:2229] T 96451ce78a6f45479178c8d596430538 P 2224f937b1b347e2931b7383c84894f2 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260807 03:38:27.020071 16154 raft_consensus.cc:2258] T 96451ce78a6f45479178c8d596430538 P 2224f937b1b347e2931b7383c84894f2 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260807 03:38:27.021663 16154 tablet_replica.cc:330] T f99d56107b6a48db9611f1ce8282d575 P 2224f937b1b347e2931b7383c84894f2: stopping tablet replica
I20260807 03:38:27.022058 16154 raft_consensus.cc:2229] T f99d56107b6a48db9611f1ce8282d575 P 2224f937b1b347e2931b7383c84894f2 [term 3 LEADER]: Raft consensus shutting down.
I20260807 03:38:27.022737 16154 raft_consensus.cc:2258] T f99d56107b6a48db9611f1ce8282d575 P 2224f937b1b347e2931b7383c84894f2 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 03:38:27.024318 16154 tablet_replica.cc:330] T 8f6f16d016d3437d8aeae4a64c42a856 P 2224f937b1b347e2931b7383c84894f2: stopping tablet replica
I20260807 03:38:27.024720 16154 raft_consensus.cc:2229] T 8f6f16d016d3437d8aeae4a64c42a856 P 2224f937b1b347e2931b7383c84894f2 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260807 03:38:27.025100 16154 raft_consensus.cc:2258] T 8f6f16d016d3437d8aeae4a64c42a856 P 2224f937b1b347e2931b7383c84894f2 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 03:38:27.026450 16154 tablet_replica.cc:330] T a41a7b13d3284dcd8b9b616eeb93c578 P 2224f937b1b347e2931b7383c84894f2: stopping tablet replica
I20260807 03:38:27.026854 16154 raft_consensus.cc:2229] T a41a7b13d3284dcd8b9b616eeb93c578 P 2224f937b1b347e2931b7383c84894f2 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260807 03:38:27.027204 16154 raft_consensus.cc:2258] T a41a7b13d3284dcd8b9b616eeb93c578 P 2224f937b1b347e2931b7383c84894f2 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 03:38:27.028774 16154 tablet_replica.cc:330] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2: stopping tablet replica
I20260807 03:38:27.029146 16154 raft_consensus.cc:2229] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 6 FOLLOWER]: Raft consensus shutting down.
I20260807 03:38:27.029472 16154 raft_consensus.cc:2258] T e3f7b02979d648108ef56a2acf13fbf0 P 2224f937b1b347e2931b7383c84894f2 [term 6 FOLLOWER]: Raft consensus is shut down!
I20260807 03:38:27.030857 16154 tablet_replica.cc:330] T 64fc83d22ad945ad890e67f9763fe413 P 2224f937b1b347e2931b7383c84894f2: stopping tablet replica
I20260807 03:38:27.031220 16154 raft_consensus.cc:2229] T 64fc83d22ad945ad890e67f9763fe413 P 2224f937b1b347e2931b7383c84894f2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260807 03:38:27.031543 16154 raft_consensus.cc:2258] T 64fc83d22ad945ad890e67f9763fe413 P 2224f937b1b347e2931b7383c84894f2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260807 03:38:27.058580 16154 tablet_server.cc:194] TabletServer@127.15.198.131:0 shutdown complete.
I20260807 03:38:27.076675 16154 master.cc:537] Master@127.15.198.190:41401 shutting down...
I20260807 03:38:27.101760 16154 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a2ac9ea50c6c4ec8a9d3d887fa5782c5 [term 1 LEADER]: Raft consensus shutting down.
I20260807 03:38:27.102334 16154 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a2ac9ea50c6c4ec8a9d3d887fa5782c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:38:27.102686 16154 tablet_replica.cc:330] T 00000000000000000000000000000000 P a2ac9ea50c6c4ec8a9d3d887fa5782c5: stopping tablet replica
I20260807 03:38:27.120595 16154 master.cc:552] Master@127.15.198.190:41401 shutdown complete.
2224f937b1b347e2931b7383c84894f2 1
3cda358d5d184a4383a9716334390022 5
6c5dca27966a4b8cadb7cc0a8e0d1676 4
2224f937b1b347e2931b7383c84894f2 4
3cda358d5d184a4383a9716334390022 3
6c5dca27966a4b8cadb7cc0a8e0d1676 3
2224f937b1b347e2931b7383c84894f2 1
3cda358d5d184a4383a9716334390022 1
6c5dca27966a4b8cadb7cc0a8e0d1676 8
2224f937b1b347e2931b7383c84894f2 3
3cda358d5d184a4383a9716334390022 3
6c5dca27966a4b8cadb7cc0a8e0d1676 4
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (28569 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20260807 03:38:27.162514 16154 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.15.198.190:40799
I20260807 03:38:27.163640 16154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:38:27.169996 17882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:38:27.170714 17884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:38:27.170837 17887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:38:27.171352 16154 server_base.cc:1034] running on GCE node
I20260807 03:38:27.172478 16154 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260807 03:38:27.172655 16154 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260807 03:38:27.172819 16154 hybrid_clock.cc:648] HybridClock initialized: now 1786073907172808 us; error 0 us; skew 500 ppm
I20260807 03:38:27.173372 16154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:38:27.179899 16154 webserver.cc:460] Webserver started at http://127.15.198.190:37709/ using document root <none> and password file <none>
I20260807 03:38:27.180436 16154 fs_manager.cc:359] Metadata directory not provided
I20260807 03:38:27.180588 16154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:38:27.180866 16154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:38:27.182106 16154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786073845599630-16154-0/minicluster-data/master-0-root/instance:
uuid: "8da2065a77024f35bdfc0dc3adb83f69"
format_stamp: "Formatted at 2026-08-07 03:38:27 on dist-test-slave-7vjv"
I20260807 03:38:27.186582 16154 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260807 03:38:27.189718 17896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:38:27.190558 16154 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 03:38:27.190771 16154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786073845599630-16154-0/minicluster-data/master-0-root
uuid: "8da2065a77024f35bdfc0dc3adb83f69"
format_stamp: "Formatted at 2026-08-07 03:38:27 on dist-test-slave-7vjv"
I20260807 03:38:27.190984 16154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786073845599630-16154-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786073845599630-16154-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786073845599630-16154-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:38:27.221982 16154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:38:27.223173 16154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:38:27.263801 16154 rpc_server.cc:307] RPC server started. Bound to: 127.15.198.190:40799
I20260807 03:38:27.263978 17976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.198.190:40799 every 8 connection(s)
I20260807 03:38:27.271077 17981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:27.285729 17981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8da2065a77024f35bdfc0dc3adb83f69: Bootstrap starting.
I20260807 03:38:27.291989 17981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8da2065a77024f35bdfc0dc3adb83f69: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:27.297489 17981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8da2065a77024f35bdfc0dc3adb83f69: No bootstrap required, opened a new log
I20260807 03:38:27.300259 17981 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8da2065a77024f35bdfc0dc3adb83f69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da2065a77024f35bdfc0dc3adb83f69" member_type: VOTER }
I20260807 03:38:27.300901 17981 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8da2065a77024f35bdfc0dc3adb83f69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:27.301213 17981 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8da2065a77024f35bdfc0dc3adb83f69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8da2065a77024f35bdfc0dc3adb83f69, State: Initialized, Role: FOLLOWER
I20260807 03:38:27.302156 17981 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8da2065a77024f35bdfc0dc3adb83f69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da2065a77024f35bdfc0dc3adb83f69" member_type: VOTER }
I20260807 03:38:27.302855 17981 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8da2065a77024f35bdfc0dc3adb83f69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:38:27.303159 17981 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8da2065a77024f35bdfc0dc3adb83f69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:38:27.303495 17981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8da2065a77024f35bdfc0dc3adb83f69 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:27.307478 17981 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8da2065a77024f35bdfc0dc3adb83f69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da2065a77024f35bdfc0dc3adb83f69" member_type: VOTER }
I20260807 03:38:27.308028 17981 leader_election.cc:302] T 00000000000000000000000000000000 P 8da2065a77024f35bdfc0dc3adb83f69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8da2065a77024f35bdfc0dc3adb83f69; no voters:
I20260807 03:38:27.309429 17981 leader_election.cc:288] T 00000000000000000000000000000000 P 8da2065a77024f35bdfc0dc3adb83f69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:38:27.309926 17984 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8da2065a77024f35bdfc0dc3adb83f69 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:27.311558 17984 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8da2065a77024f35bdfc0dc3adb83f69 [term 1 LEADER]: Becoming Leader. State: Replica: 8da2065a77024f35bdfc0dc3adb83f69, State: Running, Role: LEADER
I20260807 03:38:27.312520 17981 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8da2065a77024f35bdfc0dc3adb83f69 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:38:27.312336 17984 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8da2065a77024f35bdfc0dc3adb83f69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da2065a77024f35bdfc0dc3adb83f69" member_type: VOTER }
I20260807 03:38:27.317714 17985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8da2065a77024f35bdfc0dc3adb83f69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8da2065a77024f35bdfc0dc3adb83f69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da2065a77024f35bdfc0dc3adb83f69" member_type: VOTER } }
I20260807 03:38:27.318383 17985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8da2065a77024f35bdfc0dc3adb83f69 [sys.catalog]: This master's current role is: LEADER
I20260807 03:38:27.320724 17986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8da2065a77024f35bdfc0dc3adb83f69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8da2065a77024f35bdfc0dc3adb83f69. Latest consensus state: current_term: 1 leader_uuid: "8da2065a77024f35bdfc0dc3adb83f69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da2065a77024f35bdfc0dc3adb83f69" member_type: VOTER } }
I20260807 03:38:27.321398 17986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8da2065a77024f35bdfc0dc3adb83f69 [sys.catalog]: This master's current role is: LEADER
I20260807 03:38:27.323791 17996 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:38:27.331740 17996 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:38:27.332834 16154 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260807 03:38:27.339879 17996 catalog_manager.cc:1347] Generated new cluster ID: b8fdb2dfe33940ab9a6e004311fd1840
I20260807 03:38:27.340162 17996 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:38:27.386245 17996 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:38:27.387820 17996 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:38:27.409147 17996 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8da2065a77024f35bdfc0dc3adb83f69: Generated new TSK 0
I20260807 03:38:27.409744 17996 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:38:27.463701 16154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:38:27.469640 18013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:38:27.470975 18015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:38:27.472671 18018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:38:27.473454 16154 server_base.cc:1034] running on GCE node
I20260807 03:38:27.474439 16154 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260807 03:38:27.474658 16154 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260807 03:38:27.474822 16154 hybrid_clock.cc:648] HybridClock initialized: now 1786073907474810 us; error 0 us; skew 500 ppm
I20260807 03:38:27.475346 16154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:38:27.478029 16154 webserver.cc:460] Webserver started at http://127.15.198.129:38279/ using document root <none> and password file <none>
I20260807 03:38:27.478540 16154 fs_manager.cc:359] Metadata directory not provided
I20260807 03:38:27.478744 16154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:38:27.479007 16154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:38:27.480247 16154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786073845599630-16154-0/minicluster-data/ts-0-root/instance:
uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
format_stamp: "Formatted at 2026-08-07 03:38:27 on dist-test-slave-7vjv"
I20260807 03:38:27.484969 16154 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260807 03:38:27.488135 18023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:38:27.488891 16154 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260807 03:38:27.489172 16154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786073845599630-16154-0/minicluster-data/ts-0-root
uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
format_stamp: "Formatted at 2026-08-07 03:38:27 on dist-test-slave-7vjv"
I20260807 03:38:27.489440 16154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786073845599630-16154-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786073845599630-16154-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786073845599630-16154-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:38:27.500612 16154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:38:27.501863 16154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:38:27.503278 16154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:38:27.505563 16154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:38:27.505777 16154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:38:27.506013 16154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:38:27.506191 16154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:38:27.548548 16154 rpc_server.cc:307] RPC server started. Bound to: 127.15.198.129:36163
I20260807 03:38:27.548640 18113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.198.129:36163 every 8 connection(s)
I20260807 03:38:27.553313 16154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:38:27.561599 18123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:38:27.562858 18125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:38:27.566457 18128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:38:27.567531 16154 server_base.cc:1034] running on GCE node
I20260807 03:38:27.569530 16154 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260807 03:38:27.569782 16154 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260807 03:38:27.569994 16154 hybrid_clock.cc:648] HybridClock initialized: now 1786073907569979 us; error 0 us; skew 500 ppm
I20260807 03:38:27.570106 18115 heartbeater.cc:347] Connected to a master server at 127.15.198.190:40799
I20260807 03:38:27.570492 18115 heartbeater.cc:464] Registering TS with master...
I20260807 03:38:27.570745 16154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:38:27.571342 18115 heartbeater.cc:511] Master 127.15.198.190:40799 requested a full tablet report, sending...
I20260807 03:38:27.573331 16154 webserver.cc:460] Webserver started at http://127.15.198.130:44941/ using document root <none> and password file <none>
I20260807 03:38:27.573694 17920 ts_manager.cc:194] Registered new tserver with Master: d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163)
I20260807 03:38:27.573952 16154 fs_manager.cc:359] Metadata directory not provided
I20260807 03:38:27.574218 16154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:38:27.574514 16154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:38:27.576036 16154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786073845599630-16154-0/minicluster-data/ts-1-root/instance:
uuid: "3c4bceb836db4536a0f255fdf86abc02"
format_stamp: "Formatted at 2026-08-07 03:38:27 on dist-test-slave-7vjv"
I20260807 03:38:27.576361 17920 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39492
I20260807 03:38:27.581316 16154 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 03:38:27.585062 18136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:38:27.585743 16154 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260807 03:38:27.586043 16154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786073845599630-16154-0/minicluster-data/ts-1-root
uuid: "3c4bceb836db4536a0f255fdf86abc02"
format_stamp: "Formatted at 2026-08-07 03:38:27 on dist-test-slave-7vjv"
I20260807 03:38:27.586314 16154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786073845599630-16154-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786073845599630-16154-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786073845599630-16154-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:38:27.602135 16154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:38:27.603232 16154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:38:27.604518 16154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:38:27.606688 16154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:38:27.606931 16154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:38:27.607177 16154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:38:27.607358 16154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:38:27.647063 16154 rpc_server.cc:307] RPC server started. Bound to: 127.15.198.130:37057
I20260807 03:38:27.647183 18228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.198.130:37057 every 8 connection(s)
I20260807 03:38:27.651500 16154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:38:27.660867 18237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:38:27.661237 18236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:38:27.662770 18239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:38:27.663582 16154 server_base.cc:1034] running on GCE node
I20260807 03:38:27.664678 16154 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260807 03:38:27.664908 16154 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260807 03:38:27.665122 16154 hybrid_clock.cc:648] HybridClock initialized: now 1786073907665106 us; error 0 us; skew 500 ppm
I20260807 03:38:27.665819 16154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:38:27.665872 18231 heartbeater.cc:347] Connected to a master server at 127.15.198.190:40799
I20260807 03:38:27.666332 18231 heartbeater.cc:464] Registering TS with master...
I20260807 03:38:27.667120 18231 heartbeater.cc:511] Master 127.15.198.190:40799 requested a full tablet report, sending...
I20260807 03:38:27.668511 16154 webserver.cc:460] Webserver started at http://127.15.198.131:41591/ using document root <none> and password file <none>
I20260807 03:38:27.669072 16154 fs_manager.cc:359] Metadata directory not provided
I20260807 03:38:27.669286 16154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:38:27.669173 17920 ts_manager.cc:194] Registered new tserver with Master: 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:27.669675 16154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:38:27.670797 16154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786073845599630-16154-0/minicluster-data/ts-2-root/instance:
uuid: "66a36e1ac8e945e893668e282d394c49"
format_stamp: "Formatted at 2026-08-07 03:38:27 on dist-test-slave-7vjv"
I20260807 03:38:27.670845 17920 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39494
I20260807 03:38:27.675200 16154 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260807 03:38:27.678359 18245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:38:27.679044 16154 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 03:38:27.679335 16154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786073845599630-16154-0/minicluster-data/ts-2-root
uuid: "66a36e1ac8e945e893668e282d394c49"
format_stamp: "Formatted at 2026-08-07 03:38:27 on dist-test-slave-7vjv"
I20260807 03:38:27.679621 16154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786073845599630-16154-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786073845599630-16154-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786073845599630-16154-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:38:27.712232 16154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:38:27.713593 16154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:38:27.714859 16154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:38:27.717017 16154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:38:27.717234 16154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:38:27.717478 16154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:38:27.717662 16154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:38:27.758869 16154 rpc_server.cc:307] RPC server started. Bound to: 127.15.198.131:33021
I20260807 03:38:27.758972 18346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.198.131:33021 every 8 connection(s)
I20260807 03:38:27.773701 18347 heartbeater.cc:347] Connected to a master server at 127.15.198.190:40799
I20260807 03:38:27.774024 18347 heartbeater.cc:464] Registering TS with master...
I20260807 03:38:27.774583 18347 heartbeater.cc:511] Master 127.15.198.190:40799 requested a full tablet report, sending...
I20260807 03:38:27.776340 17920 ts_manager.cc:194] Registered new tserver with Master: 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:27.777444 16154 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.015551911s
I20260807 03:38:27.778095 17920 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39510
I20260807 03:38:27.780323 16154 test_util.cc:268] Using random seed: -1127163190
I20260807 03:38:27.804267 17920 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39518:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 03:38:27.809867 17920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 03:38:28.022074 18179 tablet_service.cc:1467] Processing CreateTablet for tablet 0890f92b53f04149998f1f1c4e7ba60d (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260807 03:38:28.023553 18179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0890f92b53f04149998f1f1c4e7ba60d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.024442 18176 tablet_service.cc:1467] Processing CreateTablet for tablet 67de6ea5223f4eefa2edd875f9dfd1bc (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260807 03:38:28.025274 18173 tablet_service.cc:1467] Processing CreateTablet for tablet 79cf89cd640d48bcbbf376c2d01e79d4 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260807 03:38:28.025830 18176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67de6ea5223f4eefa2edd875f9dfd1bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.026532 18173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79cf89cd640d48bcbbf376c2d01e79d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.023108 18177 tablet_service.cc:1467] Processing CreateTablet for tablet d8d26ed9a1e340ce8c2d127ac7fbf7f7 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260807 03:38:28.027875 18177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8d26ed9a1e340ce8c2d127ac7fbf7f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.032748 18168 tablet_service.cc:1467] Processing CreateTablet for tablet a21f43f2cbf24aeca08087d9e2fb9b2a (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260807 03:38:28.033099 18172 tablet_service.cc:1467] Processing CreateTablet for tablet 91555cf08ec8405c9d002ad0c11f8e5e (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260807 03:38:28.033993 18168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a21f43f2cbf24aeca08087d9e2fb9b2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.035281 18169 tablet_service.cc:1467] Processing CreateTablet for tablet 64ca0223f47b46fd955704eba5259370 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260807 03:38:28.036548 18169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64ca0223f47b46fd955704eba5259370. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.041213 18167 tablet_service.cc:1467] Processing CreateTablet for tablet 4cd2b7ebeed04ac6907b51b064e94894 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260807 03:38:28.042505 18167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cd2b7ebeed04ac6907b51b064e94894. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.045760 18165 tablet_service.cc:1467] Processing CreateTablet for tablet 4bd704177fca4d3bbd1b9d39a5f3ec01 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260807 03:38:28.047080 18165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bd704177fca4d3bbd1b9d39a5f3ec01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.049686 18161 tablet_service.cc:1467] Processing CreateTablet for tablet 6129789267024fac8a337070c10a472a (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260807 03:38:28.058144 18179 tablet_service.cc:1467] Processing CreateTablet for tablet 922e1d3cde8f476ea7780363c6b8693c (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260807 03:38:28.060438 18172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91555cf08ec8405c9d002ad0c11f8e5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.061096 18161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6129789267024fac8a337070c10a472a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.063755 18165 tablet_service.cc:1467] Processing CreateTablet for tablet 9a59a83a64df475ea15dd572f84024bd (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260807 03:38:28.070116 18179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 922e1d3cde8f476ea7780363c6b8693c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.072639 18165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a59a83a64df475ea15dd572f84024bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.084779 18065 tablet_service.cc:1467] Processing CreateTablet for tablet 0890f92b53f04149998f1f1c4e7ba60d (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260807 03:38:28.086032 18065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0890f92b53f04149998f1f1c4e7ba60d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.088277 18064 tablet_service.cc:1467] Processing CreateTablet for tablet d8d26ed9a1e340ce8c2d127ac7fbf7f7 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260807 03:38:28.088442 18053 tablet_service.cc:1467] Processing CreateTablet for tablet 6129789267024fac8a337070c10a472a (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260807 03:38:28.089668 18064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8d26ed9a1e340ce8c2d127ac7fbf7f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.092675 18053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6129789267024fac8a337070c10a472a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.092643 18063 tablet_service.cc:1467] Processing CreateTablet for tablet 67de6ea5223f4eefa2edd875f9dfd1bc (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260807 03:38:28.094251 18063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67de6ea5223f4eefa2edd875f9dfd1bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.097620 18061 tablet_service.cc:1467] Processing CreateTablet for tablet 79cf89cd640d48bcbbf376c2d01e79d4 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260807 03:38:28.099352 18061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79cf89cd640d48bcbbf376c2d01e79d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.099160 18177 tablet_service.cc:1467] Processing CreateTablet for tablet d416e1015a2644958b0c8f9147d403a2 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260807 03:38:28.104125 18161 tablet_service.cc:1467] Processing CreateTablet for tablet 1dbff0b8e97846ea9f567f377b986efc (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260807 03:38:28.105515 18056 tablet_service.cc:1467] Processing CreateTablet for tablet 4cd2b7ebeed04ac6907b51b064e94894 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260807 03:38:28.106760 18056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cd2b7ebeed04ac6907b51b064e94894. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.109615 18054 tablet_service.cc:1467] Processing CreateTablet for tablet 4bd704177fca4d3bbd1b9d39a5f3ec01 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260807 03:38:28.111135 18054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bd704177fca4d3bbd1b9d39a5f3ec01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.114252 18058 tablet_service.cc:1467] Processing CreateTablet for tablet 64ca0223f47b46fd955704eba5259370 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260807 03:38:28.115738 18058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64ca0223f47b46fd955704eba5259370. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.116977 18167 tablet_service.cc:1467] Processing CreateTablet for tablet b58a8dc36ec94e0c8877bdf0f901598a (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260807 03:38:28.149565 18279 tablet_service.cc:1467] Processing CreateTablet for tablet 4bd704177fca4d3bbd1b9d39a5f3ec01 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260807 03:38:28.119500 18057 tablet_service.cc:1467] Processing CreateTablet for tablet a21f43f2cbf24aeca08087d9e2fb9b2a (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260807 03:38:28.175334 18057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a21f43f2cbf24aeca08087d9e2fb9b2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.175416 18365 tablet_bootstrap.cc:492] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:28.125746 18177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d416e1015a2644958b0c8f9147d403a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.127887 18176 tablet_service.cc:1467] Processing CreateTablet for tablet 8a9b0210bce148c4a06eef3019b593c6 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260807 03:38:28.129372 18179 tablet_service.cc:1467] Processing CreateTablet for tablet 96140f11b8e74293b333cd7e760c3959 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260807 03:38:28.136075 18165 tablet_service.cc:1467] Processing CreateTablet for tablet 9a34af1cebc14f92bad043893aa1f964 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260807 03:38:28.137213 18056 tablet_service.cc:1467] Processing CreateTablet for tablet 922e1d3cde8f476ea7780363c6b8693c (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260807 03:38:28.139503 18282 tablet_service.cc:1467] Processing CreateTablet for tablet 64ca0223f47b46fd955704eba5259370 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260807 03:38:28.180570 18176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a9b0210bce148c4a06eef3019b593c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.181252 18282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64ca0223f47b46fd955704eba5259370. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.183668 18165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a34af1cebc14f92bad043893aa1f964. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.140236 18290 tablet_service.cc:1467] Processing CreateTablet for tablet 0890f92b53f04149998f1f1c4e7ba60d (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260807 03:38:28.184881 18290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0890f92b53f04149998f1f1c4e7ba60d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.185315 18365 tablet_bootstrap.cc:654] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:28.186587 18161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dbff0b8e97846ea9f567f377b986efc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.141872 18286 tablet_service.cc:1467] Processing CreateTablet for tablet d8d26ed9a1e340ce8c2d127ac7fbf7f7 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260807 03:38:28.143400 18169 tablet_service.cc:1467] Processing CreateTablet for tablet 33252bff379e4392a9a815ecdbc447a7 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260807 03:38:28.189020 18169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33252bff379e4392a9a815ecdbc447a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.190871 18179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96140f11b8e74293b333cd7e760c3959. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.143496 18285 tablet_service.cc:1467] Processing CreateTablet for tablet 67de6ea5223f4eefa2edd875f9dfd1bc (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260807 03:38:28.145081 18284 tablet_service.cc:1467] Processing CreateTablet for tablet 79cf89cd640d48bcbbf376c2d01e79d4 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260807 03:38:28.146733 18283 tablet_service.cc:1467] Processing CreateTablet for tablet 91555cf08ec8405c9d002ad0c11f8e5e (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260807 03:38:28.189038 18286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8d26ed9a1e340ce8c2d127ac7fbf7f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.192466 18167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b58a8dc36ec94e0c8877bdf0f901598a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.147346 18173 tablet_service.cc:1467] Processing CreateTablet for tablet 1b214306f53244f8b6ff2a1a9019abd1 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260807 03:38:28.193373 18284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79cf89cd640d48bcbbf376c2d01e79d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.194162 18173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b214306f53244f8b6ff2a1a9019abd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.166538 18277 tablet_service.cc:1467] Processing CreateTablet for tablet 6129789267024fac8a337070c10a472a (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260807 03:38:28.196674 18283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91555cf08ec8405c9d002ad0c11f8e5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.117475 18059 tablet_service.cc:1467] Processing CreateTablet for tablet 91555cf08ec8405c9d002ad0c11f8e5e (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260807 03:38:28.166726 18168 tablet_service.cc:1467] Processing CreateTablet for tablet d0047935053942d480774d46b081709d (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260807 03:38:28.200420 18277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6129789267024fac8a337070c10a472a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.201157 18168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0047935053942d480774d46b081709d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.204674 18285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67de6ea5223f4eefa2edd875f9dfd1bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.204769 18059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91555cf08ec8405c9d002ad0c11f8e5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.208519 18056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 922e1d3cde8f476ea7780363c6b8693c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.148729 18281 tablet_service.cc:1467] Processing CreateTablet for tablet a21f43f2cbf24aeca08087d9e2fb9b2a (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260807 03:38:28.228996 18281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a21f43f2cbf24aeca08087d9e2fb9b2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.150460 18280 tablet_service.cc:1467] Processing CreateTablet for tablet 4cd2b7ebeed04ac6907b51b064e94894 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260807 03:38:28.232909 18280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cd2b7ebeed04ac6907b51b064e94894. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.235431 18165 tablet_service.cc:1467] Processing CreateTablet for tablet c702701ec4504375977f142ffc7f0be0 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260807 03:38:28.238384 18165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c702701ec4504375977f142ffc7f0be0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.243276 18366 tablet_bootstrap.cc:492] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:28.177227 18064 tablet_service.cc:1467] Processing CreateTablet for tablet 9a59a83a64df475ea15dd572f84024bd (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260807 03:38:28.248824 18064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a59a83a64df475ea15dd572f84024bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.263139 18366 tablet_bootstrap.cc:654] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:28.264657 18279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bd704177fca4d3bbd1b9d39a5f3ec01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.266192 18172 tablet_service.cc:1467] Processing CreateTablet for tablet b2d369700aef4621b22ba058956d0894 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260807 03:38:28.268059 18176 tablet_service.cc:1467] Processing CreateTablet for tablet 2371c1d6174640aa9a9ee19732634a88 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260807 03:38:28.269285 18053 tablet_service.cc:1467] Processing CreateTablet for tablet 1dbff0b8e97846ea9f567f377b986efc (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260807 03:38:28.270172 18177 tablet_service.cc:1467] Processing CreateTablet for tablet c8c916a6b2a8442c9a05c532603c5904 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260807 03:38:28.283854 18177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8c916a6b2a8442c9a05c532603c5904. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.288689 18053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dbff0b8e97846ea9f567f377b986efc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.289783 18179 tablet_service.cc:1467] Processing CreateTablet for tablet 8ca2374bfc1e4584bd24f81b2ab5d0f1 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260807 03:38:28.255596 18059 tablet_service.cc:1467] Processing CreateTablet for tablet d416e1015a2644958b0c8f9147d403a2 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260807 03:38:28.292196 18059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d416e1015a2644958b0c8f9147d403a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.291278 18179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ca2374bfc1e4584bd24f81b2ab5d0f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.302922 18056 tablet_service.cc:1467] Processing CreateTablet for tablet b58a8dc36ec94e0c8877bdf0f901598a (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260807 03:38:28.305903 18290 tablet_service.cc:1467] Processing CreateTablet for tablet 9a59a83a64df475ea15dd572f84024bd (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260807 03:38:28.307291 18172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2d369700aef4621b22ba058956d0894. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.314015 18061 tablet_service.cc:1467] Processing CreateTablet for tablet 8a9b0210bce148c4a06eef3019b593c6 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260807 03:38:28.315204 18176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2371c1d6174640aa9a9ee19732634a88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.303712 18063 tablet_service.cc:1467] Processing CreateTablet for tablet 96140f11b8e74293b333cd7e760c3959 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260807 03:38:28.324326 18290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a59a83a64df475ea15dd572f84024bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.315196 18061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a9b0210bce148c4a06eef3019b593c6. 1 dirs total, 0 dirs full, 0 dirs failed
W20260807 03:38:28.327963 18003 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 03:38:28.327755 18169 tablet_service.cc:1467] Processing CreateTablet for tablet 79e3ecc1bf7d4ca9a7096616efc98585 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260807 03:38:28.328310 18282 tablet_service.cc:1467] Processing CreateTablet for tablet d416e1015a2644958b0c8f9147d403a2 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260807 03:38:28.329598 18063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96140f11b8e74293b333cd7e760c3959. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.330004 18365 tablet_bootstrap.cc:492] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:28.332123 18365 ts_tablet_manager.cc:1397] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.157s user 0.017s sys 0.000s
I20260807 03:38:28.334826 18365 raft_consensus.cc:348] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:28.335551 18365 raft_consensus.cc:374] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:28.335829 18365 raft_consensus.cc:729] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:28.336843 18369 tablet_bootstrap.cc:492] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:28.329598 18282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d416e1015a2644958b0c8f9147d403a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.352331 18169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79e3ecc1bf7d4ca9a7096616efc98585. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.352936 18057 tablet_service.cc:1467] Processing CreateTablet for tablet 9a34af1cebc14f92bad043893aa1f964 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260807 03:38:28.354198 18057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a34af1cebc14f92bad043893aa1f964. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.354044 18365 consensus_queue.cc:260] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:28.354367 18054 tablet_service.cc:1467] Processing CreateTablet for tablet c702701ec4504375977f142ffc7f0be0 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260807 03:38:28.355581 18054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c702701ec4504375977f142ffc7f0be0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.357476 18284 tablet_service.cc:1467] Processing CreateTablet for tablet b58a8dc36ec94e0c8877bdf0f901598a (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260807 03:38:28.358630 18284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b58a8dc36ec94e0c8877bdf0f901598a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.359850 18281 tablet_service.cc:1467] Processing CreateTablet for tablet 96140f11b8e74293b333cd7e760c3959 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260807 03:38:28.307883 18167 tablet_service.cc:1467] Processing CreateTablet for tablet 7b26fcaa55344cf98fc76dd083788d00 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260807 03:38:28.368963 18167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b26fcaa55344cf98fc76dd083788d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.368773 18277 tablet_service.cc:1467] Processing CreateTablet for tablet 1dbff0b8e97846ea9f567f377b986efc (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260807 03:38:28.369979 18277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dbff0b8e97846ea9f567f377b986efc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.373066 18285 tablet_service.cc:1467] Processing CreateTablet for tablet 8a9b0210bce148c4a06eef3019b593c6 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260807 03:38:28.373674 18369 tablet_bootstrap.cc:654] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:28.374420 18285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a9b0210bce148c4a06eef3019b593c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.379297 18168 tablet_service.cc:1467] Processing CreateTablet for tablet 28ccf467b7d24422ba092003f6b06616 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260807 03:38:28.380596 18168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28ccf467b7d24422ba092003f6b06616. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.384487 18161 tablet_service.cc:1467] Processing CreateTablet for tablet e5ea193a7604411b81684ccff437bc0a (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260807 03:38:28.385756 18161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ea193a7604411b81684ccff437bc0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.386731 18280 tablet_service.cc:1467] Processing CreateTablet for tablet 9a34af1cebc14f92bad043893aa1f964 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260807 03:38:28.388221 18283 tablet_service.cc:1467] Processing CreateTablet for tablet 922e1d3cde8f476ea7780363c6b8693c (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260807 03:38:28.388468 18173 tablet_service.cc:1467] Processing CreateTablet for tablet 071a939ead594d38851e3b554bfc4d24 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260807 03:38:28.390034 18173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 071a939ead594d38851e3b554bfc4d24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.391147 18056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b58a8dc36ec94e0c8877bdf0f901598a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.393203 18061 tablet_service.cc:1467] Processing CreateTablet for tablet 33252bff379e4392a9a815ecdbc447a7 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260807 03:38:28.394929 18061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33252bff379e4392a9a815ecdbc447a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.398865 18286 tablet_service.cc:1467] Processing CreateTablet for tablet 33252bff379e4392a9a815ecdbc447a7 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260807 03:38:28.401213 18280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a34af1cebc14f92bad043893aa1f964. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.409471 18283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 922e1d3cde8f476ea7780363c6b8693c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.417815 18369 tablet_bootstrap.cc:492] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:28.418243 18369 ts_tablet_manager.cc:1397] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.082s user 0.007s sys 0.005s
I20260807 03:38:28.420958 18369 raft_consensus.cc:348] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:28.421707 18369 raft_consensus.cc:374] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:28.422052 18369 raft_consensus.cc:729] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:28.422928 18286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33252bff379e4392a9a815ecdbc447a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.429139 18059 tablet_service.cc:1467] Processing CreateTablet for tablet 1b214306f53244f8b6ff2a1a9019abd1 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260807 03:38:28.429721 18369 consensus_queue.cc:260] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:28.430371 18059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b214306f53244f8b6ff2a1a9019abd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.451225 18179 tablet_service.cc:1467] Processing CreateTablet for tablet b15fdfae36e94837adea3c6b66864b0e (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260807 03:38:28.452526 18179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b15fdfae36e94837adea3c6b66864b0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.452591 18231 heartbeater.cc:503] Master 127.15.198.190:40799 was elected leader, sending a full tablet report...
I20260807 03:38:28.454792 18053 tablet_service.cc:1467] Processing CreateTablet for tablet b2d369700aef4621b22ba058956d0894 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260807 03:38:28.452740 18365 ts_tablet_manager.cc:1428] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.120s user 0.013s sys 0.009s
I20260807 03:38:28.448572 18281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96140f11b8e74293b333cd7e760c3959. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.464334 18282 tablet_service.cc:1467] Processing CreateTablet for tablet d0047935053942d480774d46b081709d (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260807 03:38:28.466558 18373 raft_consensus.cc:482] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:28.467252 18369 ts_tablet_manager.cc:1428] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.049s user 0.017s sys 0.005s
I20260807 03:38:28.467144 18373 raft_consensus.cc:504] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:28.468001 18347 heartbeater.cc:503] Master 127.15.198.190:40799 was elected leader, sending a full tablet report...
I20260807 03:38:28.467777 18065 tablet_service.cc:1467] Processing CreateTablet for tablet d0047935053942d480774d46b081709d (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260807 03:38:28.432241 18172 tablet_service.cc:1467] Processing CreateTablet for tablet e8d6cc12bed1425985f0504451c5e057 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260807 03:38:28.483232 18282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0047935053942d480774d46b081709d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.488855 18053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2d369700aef4621b22ba058956d0894. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.489077 18290 tablet_service.cc:1467] Processing CreateTablet for tablet 1b214306f53244f8b6ff2a1a9019abd1 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260807 03:38:28.490417 18366 tablet_bootstrap.cc:492] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:28.492376 18366 ts_tablet_manager.cc:1397] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.249s user 0.016s sys 0.004s
I20260807 03:38:28.494112 18369 tablet_bootstrap.cc:492] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:28.495417 18366 raft_consensus.cc:348] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:28.496397 18365 tablet_bootstrap.cc:492] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:28.507736 18366 raft_consensus.cc:374] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:28.508193 18366 raft_consensus.cc:729] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:28.508970 18366 consensus_queue.cc:260] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:28.504366 18277 tablet_service.cc:1467] Processing CreateTablet for tablet b2d369700aef4621b22ba058956d0894 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260807 03:38:28.516649 18284 tablet_service.cc:1467] Processing CreateTablet for tablet c702701ec4504375977f142ffc7f0be0 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260807 03:38:28.515949 18065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0047935053942d480774d46b081709d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.518831 18064 tablet_service.cc:1467] Processing CreateTablet for tablet c8c916a6b2a8442c9a05c532603c5904 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260807 03:38:28.518988 18063 tablet_service.cc:1467] Processing CreateTablet for tablet 8ca2374bfc1e4584bd24f81b2ab5d0f1 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260807 03:38:28.520026 18064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8c916a6b2a8442c9a05c532603c5904. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.514765 18369 tablet_bootstrap.cc:654] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:28.528049 18061 tablet_service.cc:1467] Processing CreateTablet for tablet 7b26fcaa55344cf98fc76dd083788d00 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260807 03:38:28.533289 18172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8d6cc12bed1425985f0504451c5e057. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.533378 18373 leader_election.cc:288] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:28.536264 18165 tablet_service.cc:1467] Processing CreateTablet for tablet 14dbf332d6394c66937326ef608dc8b5 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260807 03:38:28.537367 18165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14dbf332d6394c66937326ef608dc8b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.540602 18176 tablet_service.cc:1467] Processing CreateTablet for tablet 23b69f685f5d40edbdd2981d75bc2ad2 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260807 03:38:28.541811 18176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23b69f685f5d40edbdd2981d75bc2ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.544253 18290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b214306f53244f8b6ff2a1a9019abd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.545913 18277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2d369700aef4621b22ba058956d0894. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.546949 18057 tablet_service.cc:1467] Processing CreateTablet for tablet 79e3ecc1bf7d4ca9a7096616efc98585 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260807 03:38:28.548251 18284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c702701ec4504375977f142ffc7f0be0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.554140 18168 tablet_service.cc:1467] Processing CreateTablet for tablet af8877c3fbba4ab18765832f1390921f (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260807 03:38:28.555330 18168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af8877c3fbba4ab18765832f1390921f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.555373 18058 tablet_service.cc:1467] Processing CreateTablet for tablet 2371c1d6174640aa9a9ee19732634a88 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260807 03:38:28.566758 18063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ca2374bfc1e4584bd24f81b2ab5d0f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.566758 18370 raft_consensus.cc:482] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:28.555727 18169 tablet_service.cc:1467] Processing CreateTablet for tablet 60437eed8852484c88e84993e376670e (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260807 03:38:28.567782 18370 raft_consensus.cc:504] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:28.570277 18370 leader_election.cc:288] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:28.574898 18169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60437eed8852484c88e84993e376670e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.613338 18280 tablet_service.cc:1467] Processing CreateTablet for tablet 2371c1d6174640aa9a9ee19732634a88 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260807 03:38:28.614610 18280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2371c1d6174640aa9a9ee19732634a88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.606477 18365 tablet_bootstrap.cc:654] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:28.616657 18061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b26fcaa55344cf98fc76dd083788d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.616384 18115 heartbeater.cc:503] Master 127.15.198.190:40799 was elected leader, sending a full tablet report...
I20260807 03:38:28.615849 18285 tablet_service.cc:1467] Processing CreateTablet for tablet c8c916a6b2a8442c9a05c532603c5904 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260807 03:38:28.643412 18177 tablet_service.cc:1467] Processing CreateTablet for tablet fe8bab68c1264fc8a2c4164e0f17f9b6 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260807 03:38:28.645404 18053 tablet_service.cc:1467] Processing CreateTablet for tablet 28ccf467b7d24422ba092003f6b06616 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260807 03:38:28.646783 18285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8c916a6b2a8442c9a05c532603c5904. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.649261 18161 tablet_service.cc:1467] Processing CreateTablet for tablet 4ba8413916634e039111c838fbe1e587 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260807 03:38:28.653177 18065 tablet_service.cc:1467] Processing CreateTablet for tablet e5ea193a7604411b81684ccff437bc0a (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260807 03:38:28.660708 18282 tablet_service.cc:1467] Processing CreateTablet for tablet 7b26fcaa55344cf98fc76dd083788d00 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260807 03:38:28.655356 18059 tablet_service.cc:1467] Processing CreateTablet for tablet 071a939ead594d38851e3b554bfc4d24 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260807 03:38:28.656629 18058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2371c1d6174640aa9a9ee19732634a88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.657112 18279 tablet_service.cc:1467] Processing CreateTablet for tablet 8ca2374bfc1e4584bd24f81b2ab5d0f1 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260807 03:38:28.668457 18282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b26fcaa55344cf98fc76dd083788d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.669034 18366 ts_tablet_manager.cc:1428] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.176s user 0.024s sys 0.023s
I20260807 03:38:28.670367 18366 tablet_bootstrap.cc:492] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:28.672412 18279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ca2374bfc1e4584bd24f81b2ab5d0f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.657948 18179 tablet_service.cc:1467] Processing CreateTablet for tablet 6399a0e5977a46c8b09c11f11af795e8 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260807 03:38:28.687006 18057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79e3ecc1bf7d4ca9a7096616efc98585. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.694768 18283 tablet_service.cc:1467] Processing CreateTablet for tablet 79e3ecc1bf7d4ca9a7096616efc98585 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260807 03:38:28.699561 18167 tablet_service.cc:1467] Processing CreateTablet for tablet 6b66966247c147308a4b8488557e0bdf (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260807 03:38:28.700891 18286 tablet_service.cc:1467] Processing CreateTablet for tablet 28ccf467b7d24422ba092003f6b06616 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260807 03:38:28.705096 18054 tablet_service.cc:1467] Processing CreateTablet for tablet b15fdfae36e94837adea3c6b66864b0e (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260807 03:38:28.705302 18173 tablet_service.cc:1467] Processing CreateTablet for tablet 3433ce141bc54c5395e00fe40cf9bb69 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260807 03:38:28.709179 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ca0223f47b46fd955704eba5259370" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:28.690884 18177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe8bab68c1264fc8a2c4164e0f17f9b6. 1 dirs total, 0 dirs full, 0 dirs failed
W20260807 03:38:28.712626 18249 leader_election.cc:341] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:28.714936 18284 tablet_service.cc:1467] Processing CreateTablet for tablet 071a939ead594d38851e3b554bfc4d24 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260807 03:38:28.717365 18168 tablet_service.cc:1467] Processing CreateTablet for tablet 4ae0265e0707432980672f7cc955c571 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260807 03:38:28.696036 18283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79e3ecc1bf7d4ca9a7096616efc98585. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.697230 18053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28ccf467b7d24422ba092003f6b06616. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.722616 18173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3433ce141bc54c5395e00fe40cf9bb69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.741201 18056 tablet_service.cc:1467] Processing CreateTablet for tablet e8d6cc12bed1425985f0504451c5e057 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260807 03:38:28.742062 18369 tablet_bootstrap.cc:492] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:28.742594 18369 ts_tablet_manager.cc:1397] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.249s user 0.014s sys 0.005s
I20260807 03:38:28.732963 18284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 071a939ead594d38851e3b554bfc4d24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.754113 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ca0223f47b46fd955704eba5259370" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:38:28.755847 18248 leader_election.cc:341] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:28.756325 18248 leader_election.cc:302] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:28.757450 18373 raft_consensus.cc:2726] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:28.760205 18059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 071a939ead594d38851e3b554bfc4d24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.760929 18369 raft_consensus.cc:348] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:28.761790 18369 raft_consensus.cc:374] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:28.762179 18369 raft_consensus.cc:729] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:28.762646 18281 tablet_service.cc:1467] Processing CreateTablet for tablet e5ea193a7604411b81684ccff437bc0a (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260807 03:38:28.765578 18369 consensus_queue.cc:260] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:28.773527 18172 tablet_service.cc:1467] Processing CreateTablet for tablet 13bfb55b1d7f43dc9fb01c519c175ccf (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260807 03:38:28.774725 18369 ts_tablet_manager.cc:1428] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.032s user 0.006s sys 0.000s
I20260807 03:38:28.775625 18369 tablet_bootstrap.cc:492] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:28.775882 18165 tablet_service.cc:1467] Processing CreateTablet for tablet 1f41eb41bec8494c8a6708fbc44a22f1 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260807 03:38:28.779399 18282 tablet_service.cc:1467] Processing CreateTablet for tablet b15fdfae36e94837adea3c6b66864b0e (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260807 03:38:28.780054 18285 tablet_service.cc:1467] Processing CreateTablet for tablet e8d6cc12bed1425985f0504451c5e057 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260807 03:38:28.784327 18161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ba8413916634e039111c838fbe1e587. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.785048 18167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b66966247c147308a4b8488557e0bdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.786738 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:38:28.788173 18140 leader_election.cc:341] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:28.767005 18063 tablet_service.cc:1467] Processing CreateTablet for tablet 14dbf332d6394c66937326ef608dc8b5 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260807 03:38:28.766263 18281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ea193a7604411b81684ccff437bc0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.786682 18179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6399a0e5977a46c8b09c11f11af795e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.790594 18054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b15fdfae36e94837adea3c6b66864b0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.802136 18366 tablet_bootstrap.cc:654] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:28.808450 18282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b15fdfae36e94837adea3c6b66864b0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.802500 18065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ea193a7604411b81684ccff437bc0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.806520 18369 tablet_bootstrap.cc:654] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:28.807646 18172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13bfb55b1d7f43dc9fb01c519c175ccf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.812388 18056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8d6cc12bed1425985f0504451c5e057. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.813403 18168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ae0265e0707432980672f7cc955c571. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.816740 18285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8d6cc12bed1425985f0504451c5e057. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.820257 18286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28ccf467b7d24422ba092003f6b06616. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.826248 18165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f41eb41bec8494c8a6708fbc44a22f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.803741 18176 tablet_service.cc:1467] Processing CreateTablet for tablet a3dc635ee56b4559a87f5955bb83235a (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260807 03:38:28.842757 18176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3dc635ee56b4559a87f5955bb83235a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.843909 18167 tablet_service.cc:1467] Processing CreateTablet for tablet bb66518c65e4495e881ce860253d0801 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260807 03:38:28.849923 18365 tablet_bootstrap.cc:492] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:28.850404 18365 ts_tablet_manager.cc:1397] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.354s user 0.015s sys 0.004s
I20260807 03:38:28.850523 18064 tablet_service.cc:1467] Processing CreateTablet for tablet 23b69f685f5d40edbdd2981d75bc2ad2 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260807 03:38:28.850557 18280 tablet_service.cc:1467] Processing CreateTablet for tablet 14dbf332d6394c66937326ef608dc8b5 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260807 03:38:28.851922 18280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14dbf332d6394c66937326ef608dc8b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.852775 18177 tablet_service.cc:1467] Processing CreateTablet for tablet d24d0bbea8bd49d482ad95538ca6e631 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260807 03:38:28.853431 18365 raft_consensus.cc:348] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:28.854244 18365 raft_consensus.cc:374] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:28.854503 18365 raft_consensus.cc:729] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:28.855480 18365 consensus_queue.cc:260] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:28.857983 18365 ts_tablet_manager.cc:1428] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260807 03:38:28.858880 18365 tablet_bootstrap.cc:492] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:28.860773 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:38:28.862388 18141 leader_election.cc:341] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:28.862823 18141 leader_election.cc:302] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:28.863931 18370 raft_consensus.cc:2726] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:28.869108 18277 tablet_service.cc:1467] Processing CreateTablet for tablet 4ba8413916634e039111c838fbe1e587 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260807 03:38:28.869896 18290 tablet_service.cc:1467] Processing CreateTablet for tablet 60437eed8852484c88e84993e376670e (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260807 03:38:28.870368 18277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ba8413916634e039111c838fbe1e587. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.870555 18369 tablet_bootstrap.cc:492] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:28.871078 18290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60437eed8852484c88e84993e376670e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.869108 18283 tablet_service.cc:1467] Processing CreateTablet for tablet af8877c3fbba4ab18765832f1390921f (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260807 03:38:28.873348 18283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af8877c3fbba4ab18765832f1390921f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.873549 18063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14dbf332d6394c66937326ef608dc8b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.871079 18369 ts_tablet_manager.cc:1397] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.096s user 0.011s sys 0.002s
I20260807 03:38:28.878892 18369 raft_consensus.cc:348] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:28.879637 18369 raft_consensus.cc:374] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:28.879922 18369 raft_consensus.cc:729] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:28.881629 18064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23b69f685f5d40edbdd2981d75bc2ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.885789 18279 tablet_service.cc:1467] Processing CreateTablet for tablet 23b69f685f5d40edbdd2981d75bc2ad2 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260807 03:38:28.881778 18369 consensus_queue.cc:260] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:28.890974 18284 tablet_service.cc:1467] Processing CreateTablet for tablet 6399a0e5977a46c8b09c11f11af795e8 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260807 03:38:28.892962 18369 ts_tablet_manager.cc:1428] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.016s user 0.006s sys 0.001s
I20260807 03:38:28.894636 18169 tablet_service.cc:1467] Processing CreateTablet for tablet 4d3a25995c8341c0bf39232774e3a917 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260807 03:38:28.893311 18065 tablet_service.cc:1467] Processing CreateTablet for tablet af8877c3fbba4ab18765832f1390921f (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260807 03:38:28.900000 18369 tablet_bootstrap.cc:492] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:28.887814 18179 tablet_service.cc:1467] Processing CreateTablet for tablet c3de9873c6144d36b33a1a666691869a (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260807 03:38:28.905563 18058 tablet_service.cc:1467] Processing CreateTablet for tablet 60437eed8852484c88e84993e376670e (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260807 03:38:28.904488 18065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af8877c3fbba4ab18765832f1390921f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.910265 18058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60437eed8852484c88e84993e376670e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.912042 18054 tablet_service.cc:1467] Processing CreateTablet for tablet 4ba8413916634e039111c838fbe1e587 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260807 03:38:28.914168 18177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d24d0bbea8bd49d482ad95538ca6e631. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.916662 18279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23b69f685f5d40edbdd2981d75bc2ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.928393 18169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d3a25995c8341c0bf39232774e3a917. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.934206 18366 tablet_bootstrap.cc:492] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:28.934782 18366 ts_tablet_manager.cc:1397] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.265s user 0.012s sys 0.011s
I20260807 03:38:28.937678 18366 raft_consensus.cc:348] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:28.938731 18284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6399a0e5977a46c8b09c11f11af795e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.938776 18167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb66518c65e4495e881ce860253d0801. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.957319 18161 tablet_service.cc:1467] Processing CreateTablet for tablet 8c98c1d422654739af06653a512f7499 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260807 03:38:28.958498 18161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c98c1d422654739af06653a512f7499. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.959692 18277 tablet_service.cc:1467] Processing CreateTablet for tablet 3433ce141bc54c5395e00fe40cf9bb69 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260807 03:38:28.960920 18277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3433ce141bc54c5395e00fe40cf9bb69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.939267 18059 tablet_service.cc:1467] Processing CreateTablet for tablet 6399a0e5977a46c8b09c11f11af795e8 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260807 03:38:28.941053 18378 raft_consensus.cc:482] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:28.947224 18168 tablet_service.cc:1467] Processing CreateTablet for tablet c1476cce1786402ab984d773b26acb1b (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260807 03:38:28.964291 18281 tablet_service.cc:1467] Processing CreateTablet for tablet fe8bab68c1264fc8a2c4164e0f17f9b6 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260807 03:38:28.964429 18378 raft_consensus.cc:504] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:28.965230 18168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1476cce1786402ab984d773b26acb1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.948071 18369 tablet_bootstrap.cc:654] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:28.967674 18378 leader_election.cc:288] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:28.948674 18063 tablet_service.cc:1467] Processing CreateTablet for tablet fe8bab68c1264fc8a2c4164e0f17f9b6 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260807 03:38:28.949568 18054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ba8413916634e039111c838fbe1e587. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.950207 18365 tablet_bootstrap.cc:654] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:28.951249 18286 tablet_service.cc:1467] Processing CreateTablet for tablet 6b66966247c147308a4b8488557e0bdf (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260807 03:38:28.951488 18057 tablet_service.cc:1467] Processing CreateTablet for tablet 6b66966247c147308a4b8488557e0bdf (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260807 03:38:28.977733 18063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe8bab68c1264fc8a2c4164e0f17f9b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.965533 18281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe8bab68c1264fc8a2c4164e0f17f9b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.954799 18366 raft_consensus.cc:374] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:28.984577 18179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3de9873c6144d36b33a1a666691869a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.985849 18366 raft_consensus.cc:729] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:28.991022 18366 consensus_queue.cc:260] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:28.993500 18366 ts_tablet_manager.cc:1428] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.058s user 0.006s sys 0.000s
I20260807 03:38:28.995163 18172 tablet_service.cc:1467] Processing CreateTablet for tablet 1f5db05c008b404288067417c3f100d2 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260807 03:38:28.998385 18280 tablet_service.cc:1467] Processing CreateTablet for tablet a3dc635ee56b4559a87f5955bb83235a (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260807 03:38:28.996975 18290 tablet_service.cc:1467] Processing CreateTablet for tablet 1f41eb41bec8494c8a6708fbc44a22f1 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260807 03:38:28.999578 18365 tablet_bootstrap.cc:492] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:29.000070 18365 ts_tablet_manager.cc:1397] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.141s user 0.019s sys 0.001s
I20260807 03:38:29.001348 18053 tablet_service.cc:1467] Processing CreateTablet for tablet 4ae0265e0707432980672f7cc955c571 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260807 03:38:29.002820 18365 raft_consensus.cc:348] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.003649 18365 raft_consensus.cc:374] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:29.003938 18365 raft_consensus.cc:729] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:29.004187 18283 tablet_service.cc:1467] Processing CreateTablet for tablet 4ae0265e0707432980672f7cc955c571 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260807 03:38:29.004766 18365 consensus_queue.cc:260] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.005394 18283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ae0265e0707432980672f7cc955c571. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:29.007146 18365 ts_tablet_manager.cc:1428] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20260807 03:38:29.008070 18365 tablet_bootstrap.cc:492] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:29.012560 18059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6399a0e5977a46c8b09c11f11af795e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:29.012923 18280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3dc635ee56b4559a87f5955bb83235a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:29.017638 18286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b66966247c147308a4b8488557e0bdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:29.012564 18165 tablet_service.cc:1467] Processing CreateTablet for tablet 7afbf00d2d944c18921cc32b7c48be16 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260807 03:38:29.022015 18056 tablet_service.cc:1467] Processing CreateTablet for tablet 3433ce141bc54c5395e00fe40cf9bb69 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260807 03:38:29.026146 18057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b66966247c147308a4b8488557e0bdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:28.977133 18282 tablet_service.cc:1467] Processing CreateTablet for tablet 13bfb55b1d7f43dc9fb01c519c175ccf (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260807 03:38:29.029512 18061 tablet_service.cc:1467] Processing CreateTablet for tablet 13bfb55b1d7f43dc9fb01c519c175ccf (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260807 03:38:29.030298 18290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f41eb41bec8494c8a6708fbc44a22f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:29.030617 18061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13bfb55b1d7f43dc9fb01c519c175ccf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:29.032555 18282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13bfb55b1d7f43dc9fb01c519c175ccf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:29.038064 18169 tablet_service.cc:1467] Processing CreateTablet for tablet 349190ae153f46c9a99c29df20734d3f (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260807 03:38:29.054111 18058 tablet_service.cc:1467] Processing CreateTablet for tablet 1f41eb41bec8494c8a6708fbc44a22f1 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260807 03:38:29.055335 18058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f41eb41bec8494c8a6708fbc44a22f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:29.055992 18064 tablet_service.cc:1467] Processing CreateTablet for tablet a3dc635ee56b4559a87f5955bb83235a (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260807 03:38:29.057205 18064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3dc635ee56b4559a87f5955bb83235a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:29.057583 18165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7afbf00d2d944c18921cc32b7c48be16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:29.062188 18373 raft_consensus.cc:482] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:29.068632 18056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3433ce141bc54c5395e00fe40cf9bb69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:29.068501 18373 raft_consensus.cc:504] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.069280 18053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ae0265e0707432980672f7cc955c571. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:29.070508 18373 leader_election.cc:288] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163)
I20260807 03:38:29.072082 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:29.072441 18366 tablet_bootstrap.cc:492] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
W20260807 03:38:29.073694 18248 leader_election.cc:341] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:29.077145 18176 tablet_service.cc:1467] Processing CreateTablet for tablet 303e3bbfb1c74e1686eae36683a96314 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260807 03:38:29.078632 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:29.085840 18193 raft_consensus.cc:2454] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 0.
I20260807 03:38:29.086891 18249 leader_election.cc:302] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:29.087355 18285 tablet_service.cc:1467] Processing CreateTablet for tablet d24d0bbea8bd49d482ad95538ca6e631 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260807 03:38:29.087976 18373 raft_consensus.cc:2781] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:29.088346 18373 raft_consensus.cc:482] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:29.088723 18373 raft_consensus.cc:3037] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:29.079847 18369 tablet_bootstrap.cc:492] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:29.092031 18369 ts_tablet_manager.cc:1397] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.192s user 0.015s sys 0.004s
I20260807 03:38:29.083918 18173 tablet_service.cc:1467] Processing CreateTablet for tablet c7a9d935105d41829202c4c161ff09e3 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260807 03:38:29.098592 18285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d24d0bbea8bd49d482ad95538ca6e631. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:29.066354 18172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f5db05c008b404288067417c3f100d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:29.105895 18176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 303e3bbfb1c74e1686eae36683a96314. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:29.107853 18169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 349190ae153f46c9a99c29df20734d3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:29.119314 18373 raft_consensus.cc:504] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.111491 18365 tablet_bootstrap.cc:654] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:29.128115 18059 tablet_service.cc:1467] Processing CreateTablet for tablet bb66518c65e4495e881ce860253d0801 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260807 03:38:29.117129 18369 raft_consensus.cc:348] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.130133 18369 raft_consensus.cc:374] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:29.130462 18369 raft_consensus.cc:729] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:29.082850 18277 tablet_service.cc:1467] Processing CreateTablet for tablet bb66518c65e4495e881ce860253d0801 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260807 03:38:29.133486 18277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb66518c65e4495e881ce860253d0801. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:29.132267 18059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb66518c65e4495e881ce860253d0801. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:29.138586 18173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7a9d935105d41829202c4c161ff09e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:29.147321 18366 tablet_bootstrap.cc:654] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:29.148900 18065 tablet_service.cc:1467] Processing CreateTablet for tablet d24d0bbea8bd49d482ad95538ca6e631 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260807 03:38:29.150172 18065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d24d0bbea8bd49d482ad95538ca6e631. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:29.151420 18282 tablet_service.cc:1467] Processing CreateTablet for tablet c1476cce1786402ab984d773b26acb1b (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260807 03:38:29.157024 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:29.161916 18193 raft_consensus.cc:3037] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:29.161923 18061 tablet_service.cc:1467] Processing CreateTablet for tablet c3de9873c6144d36b33a1a666691869a (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260807 03:38:29.163273 18061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3de9873c6144d36b33a1a666691869a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:29.158803 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:29.164580 18281 tablet_service.cc:1467] Processing CreateTablet for tablet c3de9873c6144d36b33a1a666691869a (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
W20260807 03:38:29.165247 18248 leader_election.cc:341] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:29.166159 18281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3de9873c6144d36b33a1a666691869a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:29.168202 18282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1476cce1786402ab984d773b26acb1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:29.159268 18373 leader_election.cc:288] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163)
I20260807 03:38:29.174079 18056 tablet_service.cc:1467] Processing CreateTablet for tablet 4d3a25995c8341c0bf39232774e3a917 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260807 03:38:29.175712 18056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d3a25995c8341c0bf39232774e3a917. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:29.178393 18177 tablet_service.cc:1467] Processing CreateTablet for tablet 4a8b888de6f046a88c6832c234e5de79 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260807 03:38:29.179886 18177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a8b888de6f046a88c6832c234e5de79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:29.181988 18373 raft_consensus.cc:482] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:29.210968 18279 tablet_service.cc:1467] Processing CreateTablet for tablet 1f5db05c008b404288067417c3f100d2 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260807 03:38:29.188585 18290 tablet_service.cc:1467] Processing CreateTablet for tablet 8c98c1d422654739af06653a512f7499 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260807 03:38:29.191658 18373 raft_consensus.cc:504] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.216655 18365 tablet_bootstrap.cc:492] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:29.193290 18168 tablet_service.cc:1467] Processing CreateTablet for tablet 7285c5f72c354324b78146f4ed947cad (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260807 03:38:29.221359 18369 consensus_queue.cc:260] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.255393 18279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f5db05c008b404288067417c3f100d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:29.228478 18063 tablet_service.cc:1467] Processing CreateTablet for tablet c1476cce1786402ab984d773b26acb1b (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260807 03:38:29.256491 18063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1476cce1786402ab984d773b26acb1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:29.244839 18058 tablet_service.cc:1467] Processing CreateTablet for tablet 1f5db05c008b404288067417c3f100d2 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260807 03:38:29.255108 18168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7285c5f72c354324b78146f4ed947cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:29.269220 18285 tablet_service.cc:1467] Processing CreateTablet for tablet 7afbf00d2d944c18921cc32b7c48be16 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260807 03:38:29.277326 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:29.281263 18373 leader_election.cc:288] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
W20260807 03:38:29.285988 18027 leader_election.cc:341] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:29.263504 18365 ts_tablet_manager.cc:1397] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.256s user 0.016s sys 0.013s
I20260807 03:38:29.286410 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:29.294714 18290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c98c1d422654739af06653a512f7499. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:29.294793 18058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f5db05c008b404288067417c3f100d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:29.266120 18054 tablet_service.cc:1467] Processing CreateTablet for tablet 8c98c1d422654739af06653a512f7499 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
W20260807 03:38:29.296581 18249 leader_election.cc:341] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:29.292286 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:38:29.302906 18248 leader_election.cc:341] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:29.303117 18057 tablet_service.cc:1467] Processing CreateTablet for tablet 349190ae153f46c9a99c29df20734d3f (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260807 03:38:29.303369 18248 leader_election.cc:302] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:29.304528 18373 raft_consensus.cc:2726] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:29.303964 18053 tablet_service.cc:1467] Processing CreateTablet for tablet 303e3bbfb1c74e1686eae36683a96314 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260807 03:38:29.295267 18284 tablet_service.cc:1467] Processing CreateTablet for tablet 4d3a25995c8341c0bf39232774e3a917 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260807 03:38:29.313930 18285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7afbf00d2d944c18921cc32b7c48be16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:29.318495 18365 raft_consensus.cc:348] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.319221 18365 raft_consensus.cc:374] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:29.319526 18365 raft_consensus.cc:729] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:29.313735 18280 tablet_service.cc:1467] Processing CreateTablet for tablet 303e3bbfb1c74e1686eae36683a96314 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260807 03:38:29.320977 18280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 303e3bbfb1c74e1686eae36683a96314. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:29.321372 18061 tablet_service.cc:1467] Processing CreateTablet for tablet c7a9d935105d41829202c4c161ff09e3 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260807 03:38:29.324626 18283 tablet_service.cc:1467] Processing CreateTablet for tablet 349190ae153f46c9a99c29df20734d3f (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260807 03:38:29.271461 18059 tablet_service.cc:1467] Processing CreateTablet for tablet 7afbf00d2d944c18921cc32b7c48be16 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260807 03:38:29.329170 18286 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57260 (request call id 40) took 1010 ms. Trace:
W20260807 03:38:29.330293 18003 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 03:38:29.329574 18286 rpcz_store.cc:276] 0807 03:38:28.318846 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:28.950334 (+631488us) service_pool.cc:227] Handling call
0807 03:38:28.976308 (+ 25974us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0807 03:38:28.976400 (+ 92us) ts_tablet_manager.cc:676] Creating new metadata...
0807 03:38:29.183603 (+207203us) tablet_metadata.cc:641] Metadata flushed
0807 03:38:29.328559 (+144956us) tablet_replica.cc:205] Creating consensus instance
0807 03:38:29.329112 (+ 553us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 03:38:29.336601 18193 raft_consensus.cc:2454] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:29.326222 18283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 349190ae153f46c9a99c29df20734d3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:29.327162 18061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7a9d935105d41829202c4c161ff09e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:29.336581 18369 ts_tablet_manager.cc:1428] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.244s user 0.013s sys 0.020s
I20260807 03:38:29.338723 18057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 349190ae153f46c9a99c29df20734d3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:29.338779 18281 tablet_service.cc:1467] Processing CreateTablet for tablet c7a9d935105d41829202c4c161ff09e3 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260807 03:38:29.339301 18369 tablet_bootstrap.cc:492] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:29.340397 18053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 303e3bbfb1c74e1686eae36683a96314. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:29.348526 18249 leader_election.cc:302] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:29.352826 18366 tablet_bootstrap.cc:492] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:29.353266 18366 ts_tablet_manager.cc:1397] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.281s user 0.016s sys 0.022s
I20260807 03:38:29.356122 18373 raft_consensus.cc:2781] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:29.355891 18366 raft_consensus.cc:348] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.356652 18373 raft_consensus.cc:686] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:29.356757 18366 raft_consensus.cc:374] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:29.357195 18366 raft_consensus.cc:729] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:29.357472 18373 consensus_queue.cc:237] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.358018 18366 consensus_queue.cc:260] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.360940 18366 ts_tablet_manager.cc:1428] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260807 03:38:29.367064 18281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7a9d935105d41829202c4c161ff09e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:29.367823 18285 tablet_service.cc:1467] Processing CreateTablet for tablet 4a8b888de6f046a88c6832c234e5de79 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260807 03:38:29.368376 18063 tablet_service.cc:1467] Processing CreateTablet for tablet 7285c5f72c354324b78146f4ed947cad (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260807 03:38:29.371469 18059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7afbf00d2d944c18921cc32b7c48be16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:29.365957 18365 consensus_queue.cc:260] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.379338 18064 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33238 (request call id 45) took 1022 ms. Trace:
I20260807 03:38:29.381887 18063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7285c5f72c354324b78146f4ed947cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:29.383847 18065 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33238 (request call id 47) took 1012 ms. Trace:
I20260807 03:38:29.384990 18054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c98c1d422654739af06653a512f7499. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:29.380456 18366 tablet_bootstrap.cc:492] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:29.385957 18286 tablet_service.cc:1467] Processing CreateTablet for tablet 7285c5f72c354324b78146f4ed947cad (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260807 03:38:29.381263 18064 rpcz_store.cc:276] 0807 03:38:28.357352 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:29.055107 (+697755us) service_pool.cc:227] Handling call
0807 03:38:29.056771 (+ 1664us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0807 03:38:29.056860 (+ 89us) ts_tablet_manager.cc:676] Creating new metadata...
0807 03:38:29.225217 (+168357us) tablet_metadata.cc:641] Metadata flushed
0807 03:38:29.350651 (+125434us) tablet_replica.cc:205] Creating consensus instance
0807 03:38:29.379272 (+ 28621us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3820}
I20260807 03:38:29.391876 18365 ts_tablet_manager.cc:1428] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.103s user 0.005s sys 0.002s
I20260807 03:38:29.393196 18365 tablet_bootstrap.cc:492] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:29.369079 18285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a8b888de6f046a88c6832c234e5de79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:29.395649 18284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d3a25995c8341c0bf39232774e3a917. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:29.402112 18286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7285c5f72c354324b78146f4ed947cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:29.384439 18065 rpcz_store.cc:276] 0807 03:38:28.371178 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:29.135647 (+764469us) service_pool.cc:227] Handling call
0807 03:38:29.149686 (+ 14039us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0807 03:38:29.149786 (+ 100us) ts_tablet_manager.cc:676] Creating new metadata...
0807 03:38:29.333179 (+183393us) tablet_metadata.cc:641] Metadata flushed
0807 03:38:29.345228 (+ 12049us) tablet_replica.cc:205] Creating consensus instance
0807 03:38:29.383451 (+ 38223us) spinlock_profiling.cc:243] Waited 4.91 ms on lock 0x7b5800001298. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f25f851 00007fe98f25fe82 00007fe9981788b9 00007fe99708b5d8 00007fe99714f063 00007fe99714ae2f 00007fe99714b5ed 00007fe9970fc869 00007fe9906fec44 00007fe9906febd2
0807 03:38:29.383779 (+ 328us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11293952}
I20260807 03:38:29.418694 18056 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33238 (request call id 49) took 1032 ms. Trace:
I20260807 03:38:29.419096 18056 rpcz_store.cc:276] 0807 03:38:28.386375 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:29.172992 (+786617us) service_pool.cc:227] Handling call
0807 03:38:29.175029 (+ 2037us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0807 03:38:29.175285 (+ 256us) ts_tablet_manager.cc:676] Creating new metadata...
0807 03:38:29.285126 (+109841us) tablet_metadata.cc:641] Metadata flushed
0807 03:38:29.418197 (+133071us) tablet_replica.cc:205] Creating consensus instance
0807 03:38:29.418634 (+ 437us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 03:38:29.403703 18064 tablet_service.cc:1467] Processing CreateTablet for tablet 4a8b888de6f046a88c6832c234e5de79 (DEFAULT_TABLE table=test-workload [id=677ca051b50345e183d60556fc42291d]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260807 03:38:29.442062 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:29.442534 18369 tablet_bootstrap.cc:654] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:29.445350 18366 tablet_bootstrap.cc:654] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
W20260807 03:38:29.449630 18027 leader_election.cc:341] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:29.450182 18027 leader_election.cc:302] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:29.450991 18378 raft_consensus.cc:2726] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:29.456604 18378 raft_consensus.cc:482] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:29.457052 18378 raft_consensus.cc:504] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.457691 17920 catalog_manager.cc:5495] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 1 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:38:29.458582 18290 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57260 (request call id 51) took 1010 ms. Trace:
I20260807 03:38:29.458877 18378 leader_election.cc:288] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:29.459017 18290 rpcz_store.cc:276] 0807 03:38:28.447731 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:29.153169 (+705438us) service_pool.cc:227] Handling call
0807 03:38:29.263024 (+109855us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0807 03:38:29.273760 (+ 10736us) ts_tablet_manager.cc:676] Creating new metadata...
0807 03:38:29.451498 (+177738us) tablet_metadata.cc:641] Metadata flushed
0807 03:38:29.458059 (+ 6561us) tablet_replica.cc:205] Creating consensus instance
0807 03:38:29.458526 (+ 467us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 03:38:29.460467 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:29.460947 18365 tablet_bootstrap.cc:654] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:29.463359 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:29.467700 18370 raft_consensus.cc:482] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:29.468423 18373 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:29.468233 18370 raft_consensus.cc:504] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.463686 18282 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57260 (request call id 50) took 1054 ms. Trace:
I20260807 03:38:29.469056 18373 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.470470 18370 leader_election.cc:288] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:29.471639 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:38:29.473055 18141 leader_election.cc:341] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:29.473552 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:38:29.473057 18027 leader_election.cc:341] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021): Illegal state: must be running to vote when last-logged opid is not known
W20260807 03:38:29.474740 18027 leader_election.cc:341] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057): Illegal state: must be running to vote when last-logged opid is not known
W20260807 03:38:29.475309 18249 leader_election.cc:341] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:29.475281 18027 leader_election.cc:302] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:29.476142 18373 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:29.476305 18378 raft_consensus.cc:2726] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:29.469907 18282 rpcz_store.cc:276] 0807 03:38:28.409552 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:29.150555 (+741003us) service_pool.cc:227] Handling call
0807 03:38:29.167785 (+ 17230us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0807 03:38:29.167873 (+ 88us) ts_tablet_manager.cc:676] Creating new metadata...
0807 03:38:29.335771 (+167898us) tablet_metadata.cc:641] Metadata flushed
0807 03:38:29.463196 (+127425us) tablet_replica.cc:205] Creating consensus instance
0807 03:38:29.463633 (+ 437us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 03:38:29.489118 18058 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33238 (request call id 52) took 1043 ms. Trace:
I20260807 03:38:29.489339 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:38:29.490922 18248 leader_election.cc:341] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:29.491611 18248 leader_election.cc:302] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:29.492132 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:29.492820 18057 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33238 (request call id 54) took 1032 ms. Trace:
I20260807 03:38:29.493232 18373 raft_consensus.cc:2726] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:29.493186 18057 rpcz_store.cc:276] 0807 03:38:28.460671 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:29.302167 (+841496us) service_pool.cc:227] Handling call
0807 03:38:29.303906 (+ 1739us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0807 03:38:29.304001 (+ 95us) ts_tablet_manager.cc:676] Creating new metadata...
0807 03:38:29.461170 (+157169us) tablet_metadata.cc:641] Metadata flushed
0807 03:38:29.491970 (+ 30800us) tablet_replica.cc:205] Creating consensus instance
0807 03:38:29.492763 (+ 793us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 03:38:29.494597 18064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a8b888de6f046a88c6832c234e5de79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:29.489791 18058 rpcz_store.cc:276] 0807 03:38:28.445697 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:29.244007 (+798310us) service_pool.cc:227] Handling call
0807 03:38:29.274984 (+ 30977us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0807 03:38:29.290268 (+ 15284us) ts_tablet_manager.cc:676] Creating new metadata...
0807 03:38:29.362871 (+ 72603us) tablet_metadata.cc:641] Metadata flushed
0807 03:38:29.480941 (+118070us) tablet_replica.cc:205] Creating consensus instance
0807 03:38:29.489042 (+ 8101us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 03:38:29.500182 18140 leader_election.cc:341] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:29.501668 18140 leader_election.cc:302] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:29.502506 18370 raft_consensus.cc:2726] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:29.502717 18365 tablet_bootstrap.cc:492] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:29.503263 18365 ts_tablet_manager.cc:1397] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.110s user 0.014s sys 0.019s
I20260807 03:38:29.506156 18365 raft_consensus.cc:348] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.506870 18365 raft_consensus.cc:374] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:29.507166 18365 raft_consensus.cc:729] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:29.508536 18365 consensus_queue.cc:260] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.510804 18365 ts_tablet_manager.cc:1428] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260807 03:38:29.512305 18365 tablet_bootstrap.cc:492] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:29.517581 18059 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33238 (request call id 53) took 1066 ms. Trace:
I20260807 03:38:29.518271 18365 tablet_bootstrap.cc:654] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:29.519713 18063 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33238 (request call id 57) took 1026 ms. Trace:
I20260807 03:38:29.520181 18063 rpcz_store.cc:276] 0807 03:38:28.493140 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:29.367499 (+874359us) service_pool.cc:227] Handling call
0807 03:38:29.369005 (+ 1506us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0807 03:38:29.369128 (+ 123us) ts_tablet_manager.cc:676] Creating new metadata...
0807 03:38:29.446297 (+ 77169us) tablet_metadata.cc:641] Metadata flushed
0807 03:38:29.519174 (+ 72877us) tablet_replica.cc:205] Creating consensus instance
0807 03:38:29.519658 (+ 484us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 03:38:29.518018 18059 rpcz_store.cc:276] 0807 03:38:28.451043 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:29.270353 (+819310us) service_pool.cc:227] Handling call
0807 03:38:29.330913 (+ 60560us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0807 03:38:29.331036 (+ 123us) ts_tablet_manager.cc:676] Creating new metadata...
0807 03:38:29.436377 (+105341us) tablet_metadata.cc:641] Metadata flushed
0807 03:38:29.488809 (+ 52432us) tablet_replica.cc:205] Creating consensus instance
0807 03:38:29.517521 (+ 28712us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 03:38:29.521793 18348 tablet.cc:2369] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:38:29.533404 18279 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57260 (request call id 52) took 1085 ms. Trace:
I20260807 03:38:29.533717 18279 rpcz_store.cc:276] 0807 03:38:28.448061 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:29.210038 (+761977us) service_pool.cc:227] Handling call
0807 03:38:29.253764 (+ 43726us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0807 03:38:29.254907 (+ 1143us) ts_tablet_manager.cc:676] Creating new metadata...
0807 03:38:29.311003 (+ 56096us) tablet_metadata.cc:641] Metadata flushed
0807 03:38:29.531873 (+220870us) tablet_replica.cc:205] Creating consensus instance
0807 03:38:29.533336 (+ 1463us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 03:38:29.532856 18283 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57260 (request call id 54) took 1070 ms. Trace:
I20260807 03:38:29.534729 18370 raft_consensus.cc:482] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:29.534909 18283 rpcz_store.cc:276] 0807 03:38:28.462559 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:29.265940 (+803381us) service_pool.cc:227] Handling call
0807 03:38:29.325602 (+ 59662us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0807 03:38:29.325777 (+ 175us) ts_tablet_manager.cc:676] Creating new metadata...
0807 03:38:29.437954 (+112177us) tablet_metadata.cc:641] Metadata flushed
0807 03:38:29.525274 (+ 87320us) tablet_replica.cc:205] Creating consensus instance
0807 03:38:29.532598 (+ 7324us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":652}
I20260807 03:38:29.535213 18370 raft_consensus.cc:504] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.537289 18370 leader_election.cc:288] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:29.537443 18284 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57260 (request call id 49) took 1146 ms. Trace:
I20260807 03:38:29.538172 18054 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33238 (request call id 51) took 1124 ms. Trace:
I20260807 03:38:29.538101 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:29.538874 18082 raft_consensus.cc:2454] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 0.
I20260807 03:38:29.538791 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:29.540189 18140 leader_election.cc:302] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
W20260807 03:38:29.540751 18141 leader_election.cc:341] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:29.541193 18369 tablet_bootstrap.cc:492] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:29.541098 18054 rpcz_store.cc:276] 0807 03:38:28.413643 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:29.235099 (+821456us) service_pool.cc:227] Handling call
0807 03:38:29.350670 (+115571us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0807 03:38:29.366181 (+ 15511us) ts_tablet_manager.cc:676] Creating new metadata...
0807 03:38:29.504186 (+138005us) tablet_metadata.cc:641] Metadata flushed
0807 03:38:29.537722 (+ 33536us) tablet_replica.cc:205] Creating consensus instance
0807 03:38:29.538122 (+ 400us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 03:38:29.541689 18370 raft_consensus.cc:2781] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:29.542259 18370 raft_consensus.cc:482] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:29.542677 18370 raft_consensus.cc:3037] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:29.541762 18369 ts_tablet_manager.cc:1397] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.203s user 0.028s sys 0.026s
I20260807 03:38:29.543785 18365 tablet_bootstrap.cc:492] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:29.544289 18365 ts_tablet_manager.cc:1397] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.032s user 0.012s sys 0.007s
I20260807 03:38:29.537801 18284 rpcz_store.cc:276] 0807 03:38:28.391074 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:29.136868 (+745794us) service_pool.cc:227] Handling call
0807 03:38:29.312859 (+175991us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0807 03:38:29.312952 (+ 93us) ts_tablet_manager.cc:676] Creating new metadata...
0807 03:38:29.497833 (+184881us) tablet_metadata.cc:641] Metadata flushed
0807 03:38:29.536966 (+ 39133us) tablet_replica.cc:205] Creating consensus instance
0807 03:38:29.537391 (+ 425us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 03:38:29.545930 18369 raft_consensus.cc:348] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.547240 18369 raft_consensus.cc:374] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:29.547029 18365 raft_consensus.cc:348] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.547705 18369 raft_consensus.cc:729] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:29.547880 18365 raft_consensus.cc:374] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:29.549633 18369 consensus_queue.cc:260] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.552371 18369 ts_tablet_manager.cc:1428] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260807 03:38:29.552251 18370 raft_consensus.cc:504] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.555184 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:29.555584 18064 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33238 (request call id 58) took 1047 ms. Trace:
I20260807 03:38:29.555827 18312 raft_consensus.cc:3037] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:29.541715 18366 tablet_bootstrap.cc:492] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:29.558291 18366 ts_tablet_manager.cc:1397] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.178s user 0.026s sys 0.032s
I20260807 03:38:29.559113 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:29.559749 18082 raft_consensus.cc:3037] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:29.556005 18064 rpcz_store.cc:276] 0807 03:38:28.508479 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:29.402557 (+894078us) service_pool.cc:227] Handling call
0807 03:38:29.448848 (+ 46291us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0807 03:38:29.483736 (+ 34888us) ts_tablet_manager.cc:676] Creating new metadata...
0807 03:38:29.534226 (+ 50490us) tablet_metadata.cc:641] Metadata flushed
0807 03:38:29.555086 (+ 20860us) tablet_replica.cc:205] Creating consensus instance
0807 03:38:29.555521 (+ 435us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 03:38:29.561710 18366 raft_consensus.cc:348] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.562546 18366 raft_consensus.cc:374] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:29.562978 18312 raft_consensus.cc:2454] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:29.564169 18141 leader_election.cc:302] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:29.548887 18365 raft_consensus.cc:729] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:29.562950 18366 raft_consensus.cc:729] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:29.569691 18082 raft_consensus.cc:2454] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:29.569305 18365 consensus_queue.cc:260] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.569540 18366 consensus_queue.cc:260] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.570806 18378 raft_consensus.cc:482] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:29.572504 18378 raft_consensus.cc:504] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.571762 18369 tablet_bootstrap.cc:492] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:29.574577 18418 raft_consensus.cc:2781] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:29.575194 18378 leader_election.cc:288] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:29.575701 18370 leader_election.cc:288] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:29.579052 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:29.579902 18193 raft_consensus.cc:2454] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 0.
I20260807 03:38:29.580289 18369 tablet_bootstrap.cc:654] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:29.581202 18027 leader_election.cc:302] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:29.582216 18378 raft_consensus.cc:2781] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:29.582376 18365 ts_tablet_manager.cc:1428] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.038s user 0.010s sys 0.002s
I20260807 03:38:29.582583 18378 raft_consensus.cc:482] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:29.582954 18378 raft_consensus.cc:3037] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:29.583416 18365 tablet_bootstrap.cc:492] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:29.587157 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:38:29.588603 18027 leader_election.cc:341] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:29.591536 18365 tablet_bootstrap.cc:654] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:29.595046 18378 raft_consensus.cc:504] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.597972 18378 leader_election.cc:288] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:29.598507 18378 raft_consensus.cc:482] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:29.598927 18369 tablet_bootstrap.cc:492] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:29.599123 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:29.598901 18378 raft_consensus.cc:504] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.599530 18369 ts_tablet_manager.cc:1397] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.028s user 0.010s sys 0.007s
I20260807 03:38:29.599730 18193 raft_consensus.cc:3037] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:29.601001 18378 leader_election.cc:288] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:29.602079 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:38:29.603196 18027 leader_election.cc:341] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:29.604707 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:29.605572 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:38:29.606065 18027 leader_election.cc:341] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:29.606261 18311 raft_consensus.cc:2454] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 0.
I20260807 03:38:29.607159 18193 raft_consensus.cc:2454] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:38:29.607133 18027 leader_election.cc:302] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:38:29.607867 18418 raft_consensus.cc:686] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:29.607921 18378 raft_consensus.cc:2781] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:29.608659 18378 raft_consensus.cc:482] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:29.609542 18378 raft_consensus.cc:3037] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:29.608752 18027 leader_election.cc:302] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:29.610590 18418 consensus_queue.cc:237] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.611207 18425 raft_consensus.cc:2781] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:29.611675 18425 raft_consensus.cc:686] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:38:29.611670 18366 ts_tablet_manager.cc:1428] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.053s user 0.007s sys 0.006s
I20260807 03:38:29.612551 18425 consensus_queue.cc:237] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.616436 18366 tablet_bootstrap.cc:492] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:29.623425 18369 raft_consensus.cc:348] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.627291 18369 raft_consensus.cc:374] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:29.627619 18369 raft_consensus.cc:729] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:29.628322 18369 consensus_queue.cc:260] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.634665 18369 ts_tablet_manager.cc:1428] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.035s user 0.005s sys 0.004s
I20260807 03:38:29.635639 18369 tablet_bootstrap.cc:492] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:29.638497 18378 raft_consensus.cc:504] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.641119 18378 leader_election.cc:288] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:29.641741 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:29.641780 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:29.642513 18311 raft_consensus.cc:3037] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
W20260807 03:38:29.643457 18027 leader_election.cc:341] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:29.644544 18366 tablet_bootstrap.cc:654] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:29.646435 18369 tablet_bootstrap.cc:654] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:29.647807 18365 tablet_bootstrap.cc:492] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:29.649372 18365 ts_tablet_manager.cc:1397] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.066s user 0.024s sys 0.018s
I20260807 03:38:29.652643 18311 raft_consensus.cc:2454] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:38:29.653946 18027 leader_election.cc:302] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:38:29.655184 18365 raft_consensus.cc:348] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.656368 18365 raft_consensus.cc:374] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:29.656802 18365 raft_consensus.cc:729] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:29.658222 18425 raft_consensus.cc:2781] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:29.658890 18425 raft_consensus.cc:686] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 1 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:38:29.659730 18425 consensus_queue.cc:237] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.657747 18365 consensus_queue.cc:260] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.664516 18365 ts_tablet_manager.cc:1428] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.014s user 0.009s sys 0.000s
I20260807 03:38:29.665750 18365 tablet_bootstrap.cc:492] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:29.667383 17915 catalog_manager.cc:5495] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 1 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:29.674257 18365 tablet_bootstrap.cc:654] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:29.681398 17920 catalog_manager.cc:5495] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 0 to 1, leader changed from <none> to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 1 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:29.696240 18365 tablet_bootstrap.cc:492] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:29.696969 18365 ts_tablet_manager.cc:1397] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.031s user 0.017s sys 0.008s
I20260807 03:38:29.700647 18365 raft_consensus.cc:348] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.701572 18365 raft_consensus.cc:374] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:29.701911 18365 raft_consensus.cc:729] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:29.702837 18365 consensus_queue.cc:260] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.705113 18369 tablet_bootstrap.cc:492] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:29.705636 18369 ts_tablet_manager.cc:1397] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.070s user 0.019s sys 0.014s
I20260807 03:38:29.706848 18366 tablet_bootstrap.cc:492] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:29.708561 18369 raft_consensus.cc:348] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.709375 18369 raft_consensus.cc:374] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:29.709069 18366 ts_tablet_manager.cc:1397] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.095s user 0.031s sys 0.026s
I20260807 03:38:29.709678 17920 catalog_manager.cc:5495] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 0 to 1, leader changed from <none> to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 1 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:29.710378 18365 ts_tablet_manager.cc:1428] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20260807 03:38:29.711246 18365 tablet_bootstrap.cc:492] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:29.712283 18366 raft_consensus.cc:348] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.713028 18366 raft_consensus.cc:374] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:29.713302 18366 raft_consensus.cc:729] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:29.709694 18369 raft_consensus.cc:729] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:29.714027 18366 consensus_queue.cc:260] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.714339 18369 consensus_queue.cc:260] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.716666 18369 ts_tablet_manager.cc:1428] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.011s user 0.005s sys 0.003s
I20260807 03:38:29.716717 18366 ts_tablet_manager.cc:1428] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260807 03:38:29.717602 18369 tablet_bootstrap.cc:492] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:29.717722 18366 tablet_bootstrap.cc:492] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:29.729421 18365 tablet_bootstrap.cc:654] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:29.729429 18366 tablet_bootstrap.cc:654] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:29.729699 18369 tablet_bootstrap.cc:654] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:29.734545 18369 tablet_bootstrap.cc:492] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:29.734941 18369 ts_tablet_manager.cc:1397] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.004s
I20260807 03:38:29.735278 18366 tablet_bootstrap.cc:492] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:29.735659 18366 ts_tablet_manager.cc:1397] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.010s
I20260807 03:38:29.735783 18365 tablet_bootstrap.cc:492] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:29.736220 18365 ts_tablet_manager.cc:1397] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.007s
I20260807 03:38:29.737280 18369 raft_consensus.cc:348] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.737960 18369 raft_consensus.cc:374] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:29.737859 18366 raft_consensus.cc:348] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.738308 18369 raft_consensus.cc:729] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:29.738631 18366 raft_consensus.cc:374] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:29.738540 18365 raft_consensus.cc:348] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.738968 18366 raft_consensus.cc:729] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:29.739233 18365 raft_consensus.cc:374] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:29.739006 18369 consensus_queue.cc:260] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.739518 18365 raft_consensus.cc:729] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:29.739813 18366 consensus_queue.cc:260] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.741679 18366 ts_tablet_manager.cc:1428] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260807 03:38:29.742436 18369 ts_tablet_manager.cc:1428] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260807 03:38:29.742764 18366 tablet_bootstrap.cc:492] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:29.743360 18365 consensus_queue.cc:260] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.745594 18365 ts_tablet_manager.cc:1428] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260807 03:38:29.746843 18365 tablet_bootstrap.cc:492] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:29.750609 18369 tablet_bootstrap.cc:492] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:29.751971 18365 tablet_bootstrap.cc:654] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:29.756778 18369 tablet_bootstrap.cc:654] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:29.760938 18365 tablet_bootstrap.cc:492] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:29.761348 18365 ts_tablet_manager.cc:1397] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.015s user 0.013s sys 0.000s
I20260807 03:38:29.763746 18365 raft_consensus.cc:348] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.764452 18365 raft_consensus.cc:374] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:29.764616 18366 tablet_bootstrap.cc:654] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:29.764752 18365 raft_consensus.cc:729] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:29.765478 18365 consensus_queue.cc:260] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.766075 18369 tablet_bootstrap.cc:492] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:29.766492 18369 ts_tablet_manager.cc:1397] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.000s
I20260807 03:38:29.769271 18369 raft_consensus.cc:348] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.769975 18369 raft_consensus.cc:374] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:29.770269 18369 raft_consensus.cc:729] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:29.770999 18369 consensus_queue.cc:260] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.773047 18369 ts_tablet_manager.cc:1428] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20260807 03:38:29.773072 18365 ts_tablet_manager.cc:1428] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260807 03:38:29.774066 18369 tablet_bootstrap.cc:492] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:29.774151 18365 tablet_bootstrap.cc:492] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:29.779768 18369 tablet_bootstrap.cc:654] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:29.780239 18365 tablet_bootstrap.cc:654] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:29.781327 18373 raft_consensus.cc:482] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:29.781919 18373 raft_consensus.cc:504] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.785383 18373 leader_election.cc:288] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:29.790397 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:29.791232 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:29.791276 18193 raft_consensus.cc:2454] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 0.
W20260807 03:38:29.792862 18248 leader_election.cc:341] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:29.793563 18249 leader_election.cc:302] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:29.795075 18373 raft_consensus.cc:2781] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:29.795548 18373 raft_consensus.cc:482] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:29.795990 18373 raft_consensus.cc:3037] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:29.806763 18373 raft_consensus.cc:504] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.807380 18418 raft_consensus.cc:482] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:29.807859 18418 raft_consensus.cc:504] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.810333 18418 leader_election.cc:288] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:29.811978 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:38:29.813685 18141 leader_election.cc:341] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:29.814582 18369 tablet_bootstrap.cc:492] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:29.814935 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:29.815162 18369 ts_tablet_manager.cc:1397] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.041s user 0.012s sys 0.004s
I20260807 03:38:29.815649 18082 raft_consensus.cc:2454] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 0.
I20260807 03:38:29.816892 18140 leader_election.cc:302] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:29.818068 18418 raft_consensus.cc:2781] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:29.818430 18418 raft_consensus.cc:482] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:29.818810 18418 raft_consensus.cc:3037] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:29.818571 18369 raft_consensus.cc:348] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.819576 18369 raft_consensus.cc:374] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:29.821112 18373 leader_election.cc:288] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:29.819947 18369 raft_consensus.cc:729] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:29.811928 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:29.825554 18369 consensus_queue.cc:260] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.826207 18193 raft_consensus.cc:3037] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:29.826864 18418 raft_consensus.cc:504] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.829277 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:29.830153 18365 tablet_bootstrap.cc:492] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:29.830324 18369 ts_tablet_manager.cc:1428] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20260807 03:38:29.830595 18365 ts_tablet_manager.cc:1397] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.057s user 0.024s sys 0.020s
W20260807 03:38:29.830636 18248 leader_election.cc:341] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:29.831175 18369 tablet_bootstrap.cc:492] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:29.833657 18365 raft_consensus.cc:348] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.834398 18365 raft_consensus.cc:374] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:29.834668 18365 raft_consensus.cc:729] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:29.835273 18365 consensus_queue.cc:260] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.836488 18193 raft_consensus.cc:2454] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:29.837546 18365 ts_tablet_manager.cc:1428] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260807 03:38:29.838691 18365 tablet_bootstrap.cc:492] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:29.838578 18249 leader_election.cc:302] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:29.839767 18373 raft_consensus.cc:2781] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:29.840297 18373 raft_consensus.cc:686] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:29.841183 18373 consensus_queue.cc:237] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.843497 18422 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:29.843995 18422 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.845409 18365 tablet_bootstrap.cc:654] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:29.846215 18422 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:29.848479 18369 tablet_bootstrap.cc:654] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:29.850382 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:29.851261 18311 raft_consensus.cc:2454] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 0.
I20260807 03:38:29.853466 18141 leader_election.cc:302] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:29.854755 18422 raft_consensus.cc:2781] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:29.855217 18422 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:29.855611 18422 raft_consensus.cc:3037] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:29.864506 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:29.865422 18082 raft_consensus.cc:2454] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 0.
I20260807 03:38:29.881278 18370 raft_consensus.cc:482] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:29.881979 18370 raft_consensus.cc:504] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:29.886271 18248 consensus_peers.cc:487] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 -> Peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Couldn't send request to peer d9716e2bf06442f5b3b0e81087c53c4d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260807 03:38:29.890201 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:29.891049 18311 raft_consensus.cc:2454] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 0.
I20260807 03:38:29.892297 18370 leader_election.cc:288] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:29.893359 18141 leader_election.cc:302] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:29.916247 18370 raft_consensus.cc:2781] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:29.916877 18370 raft_consensus.cc:482] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:29.917404 18370 raft_consensus.cc:3037] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:29.920547 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:29.921561 18082 raft_consensus.cc:2454] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 0.
I20260807 03:38:29.960116 18418 leader_election.cc:288] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:29.965907 18422 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.968417 18369 tablet_bootstrap.cc:492] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:29.969030 18369 ts_tablet_manager.cc:1397] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.138s user 0.010s sys 0.002s
I20260807 03:38:29.973847 18422 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:29.974659 18369 raft_consensus.cc:348] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.975421 18369 raft_consensus.cc:374] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:29.975745 18369 raft_consensus.cc:729] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:29.975628 18370 raft_consensus.cc:504] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.976573 18369 consensus_queue.cc:260] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:29.978286 18370 leader_election.cc:288] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:29.979758 18369 ts_tablet_manager.cc:1428] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260807 03:38:29.980782 18369 tablet_bootstrap.cc:492] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:29.981062 18366 tablet_bootstrap.cc:492] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:29.981745 18366 ts_tablet_manager.cc:1397] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.239s user 0.057s sys 0.052s
I20260807 03:38:29.981973 18365 tablet_bootstrap.cc:492] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:29.982409 18365 ts_tablet_manager.cc:1397] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.144s user 0.049s sys 0.057s
I20260807 03:38:29.984853 18366 raft_consensus.cc:348] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:30.047679 18366 raft_consensus.cc:374] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:29.995285 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:29.995543 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:29.996004 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:30.046243 18373 raft_consensus.cc:482] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:30.050455 18080 raft_consensus.cc:3037] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:30.050495 18373 raft_consensus.cc:504] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:30.052191 18370 raft_consensus.cc:482] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:30.052632 18373 leader_election.cc:288] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:30.053182 18373 raft_consensus.cc:482] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:30.053576 18373 raft_consensus.cc:504] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:30.052654 18370 raft_consensus.cc:504] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:30.055488 18373 leader_election.cc:288] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:30.055943 18373 raft_consensus.cc:482] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:30.056345 18373 raft_consensus.cc:504] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:30.057789 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ca0223f47b46fd955704eba5259370" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:30.058215 18373 leader_election.cc:288] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:30.058707 18373 consensus_queue.cc:1035] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:30.059113 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:30.059266 18370 leader_election.cc:288] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:38:30.059223 18140 leader_election.cc:341] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:30.048035 18365 raft_consensus.cc:348] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:30.061091 18365 raft_consensus.cc:374] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:30.048030 18366 raft_consensus.cc:729] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:30.061512 18365 raft_consensus.cc:729] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:30.060211 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:30.062673 18365 consensus_queue.cc:260] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:30.061878 18081 raft_consensus.cc:3037] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:30.064774 18365 ts_tablet_manager.cc:1428] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.082s user 0.009s sys 0.002s
I20260807 03:38:30.066768 18365 tablet_bootstrap.cc:492] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:30.061415 18082 raft_consensus.cc:3037] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:30.067595 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:30.068315 18366 consensus_queue.cc:260] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:30.069818 18248 leader_election.cc:341] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:30.069774 18075 raft_consensus.cc:2454] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 0.
I20260807 03:38:30.070528 18366 ts_tablet_manager.cc:1428] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.088s user 0.009s sys 0.001s
I20260807 03:38:30.071900 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:30.072535 18193 raft_consensus.cc:2454] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 0.
I20260807 03:38:30.074440 18366 tablet_bootstrap.cc:492] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:30.076164 18369 tablet_bootstrap.cc:654] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:30.080921 18248 leader_election.cc:302] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:30.082129 18466 raft_consensus.cc:2781] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:30.082592 18080 raft_consensus.cc:2454] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:30.085318 18466 raft_consensus.cc:482] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:30.085721 18466 raft_consensus.cc:3037] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:30.087213 18140 leader_election.cc:302] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:30.086064 17914 catalog_manager.cc:5495] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 1 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:38:30.105947 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:30.106806 18191 raft_consensus.cc:2454] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 0.
I20260807 03:38:30.107016 18082 raft_consensus.cc:2454] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
W20260807 03:38:30.109068 18027 consensus_peers.cc:487] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d -> Peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057): Couldn't send request to peer 3c4bceb836db4536a0f255fdf86abc02. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260807 03:38:30.110167 18140 leader_election.cc:302] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:30.111285 18422 raft_consensus.cc:2781] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:30.111932 18422 raft_consensus.cc:686] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:30.128047 18422 consensus_queue.cc:237] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:30.097146 18370 raft_consensus.cc:2781] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:30.133553 18081 raft_consensus.cc:2454] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:30.136284 18140 leader_election.cc:302] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:30.137404 18418 raft_consensus.cc:2781] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:30.137972 18418 raft_consensus.cc:686] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:30.139978 18249 leader_election.cc:302] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:30.140218 18311 raft_consensus.cc:3037] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:30.140888 18465 raft_consensus.cc:2781] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:30.141283 18465 raft_consensus.cc:482] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:30.141661 18465 raft_consensus.cc:3037] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:30.140461 18418 consensus_queue.cc:237] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:30.144229 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:30.153270 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:30.157681 18193 raft_consensus.cc:2379] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 1: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:38:30.159587 18249 leader_election.cc:302] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:30.161492 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:30.162060 18312 raft_consensus.cc:3037] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:30.164273 18373 raft_consensus.cc:2726] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:30.150974 18370 raft_consensus.cc:686] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:30.164209 18465 raft_consensus.cc:504] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:30.179039 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:30.181942 18312 raft_consensus.cc:2454] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:30.182962 18366 tablet_bootstrap.cc:654] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
W20260807 03:38:30.184623 18248 leader_election.cc:341] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:30.185279 18369 tablet_bootstrap.cc:492] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:30.188400 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ca0223f47b46fd955704eba5259370" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:38:30.188832 18141 leader_election.cc:341] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:30.189098 18311 raft_consensus.cc:2454] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 0.
I20260807 03:38:30.190426 18425 consensus_queue.cc:1035] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:30.189844 18418 consensus_queue.cc:1035] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260807 03:38:30.192036 18466 raft_consensus.cc:504] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:30.192991 18141 leader_election.cc:302] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:30.193022 18422 consensus_queue.cc:1035] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 03:38:30.194202 18471 raft_consensus.cc:2781] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:30.201582 18471 raft_consensus.cc:482] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:30.311762 18471 raft_consensus.cc:3037] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:30.202386 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:30.312664 18081 raft_consensus.cc:3037] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:30.311429 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:30.313599 18312 raft_consensus.cc:3037] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
W20260807 03:38:30.308979 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 116067us
I20260807 03:38:30.315765 18365 tablet_bootstrap.cc:654] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:30.255893 18425 consensus_queue.cc:1035] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.061s
I20260807 03:38:30.231243 18422 raft_consensus.cc:482] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:30.319250 18422 raft_consensus.cc:504] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:30.308825 18369 ts_tablet_manager.cc:1397] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.328s user 0.024s sys 0.020s
I20260807 03:38:30.255278 18465 leader_election.cc:288] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:30.326741 18402 raft_consensus.cc:482] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:30.328444 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:30.316809 18478 consensus_queue.cc:1035] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260807 03:38:30.280716 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:30.325862 18466 leader_election.cc:288] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:30.296705 18370 consensus_queue.cc:237] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:30.346764 18193 raft_consensus.cc:3037] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
W20260807 03:38:30.354019 18140 leader_election.cc:341] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:30.350541 18422 leader_election.cc:288] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:30.344825 18402 raft_consensus.cc:504] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:30.352488 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:30.369580 18366 tablet_bootstrap.cc:492] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:30.498731 18366 ts_tablet_manager.cc:1397] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.424s user 0.028s sys 0.059s
I20260807 03:38:30.498795 18193 raft_consensus.cc:2454] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:30.500278 18249 leader_election.cc:302] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:30.374424 18369 raft_consensus.cc:348] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:30.502516 18369 raft_consensus.cc:374] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:30.502857 18369 raft_consensus.cc:729] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:30.503496 18369 consensus_queue.cc:260] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:30.497684 18479 log.cc:924] Time spent T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.074s user 0.009s sys 0.018s
W20260807 03:38:30.489849 18003 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 03:38:30.489512 17914 catalog_manager.cc:5495] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 1 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:30.511161 18483 raft_consensus.cc:2781] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:30.521345 18483 raft_consensus.cc:686] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:30.522567 18483 consensus_queue.cc:237] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:30.423664 18465 consensus_queue.cc:1035] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.089s
W20260807 03:38:30.526895 18248 consensus_peers.cc:487] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 -> Peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Couldn't send request to peer d9716e2bf06442f5b3b0e81087c53c4d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20260807 03:38:30.527392 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 96679.1us
I20260807 03:38:30.513612 18081 raft_consensus.cc:2454] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:30.515816 18366 raft_consensus.cc:348] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:30.540731 18366 raft_consensus.cc:374] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:30.541106 18366 raft_consensus.cc:729] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:30.541872 18366 consensus_queue.cc:260] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:30.409273 18471 raft_consensus.cc:504] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:30.519320 17914 catalog_manager.cc:5495] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 1 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:30.503695 18191 raft_consensus.cc:3037] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:30.393927 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:30.409178 18312 raft_consensus.cc:2454] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:30.517990 18369 ts_tablet_manager.cc:1428] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.196s user 0.010s sys 0.013s
I20260807 03:38:30.550535 18369 tablet_bootstrap.cc:492] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:30.548960 18366 ts_tablet_manager.cc:1428] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.050s user 0.006s sys 0.002s
I20260807 03:38:30.554458 18366 tablet_bootstrap.cc:492] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
W20260807 03:38:30.564471 18141 leader_election.cc:341] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:30.565135 18141 leader_election.cc:302] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:30.592948 18465 raft_consensus.cc:482] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:30.593639 18465 raft_consensus.cc:504] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:30.609261 18402 leader_election.cc:288] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:30.610347 18471 leader_election.cc:288] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:38:30.635882 18488 log.cc:924] Time spent T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.082s user 0.000s sys 0.001s
I20260807 03:38:30.583185 18248 leader_election.cc:302] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:30.655504 18369 tablet_bootstrap.cc:654] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:30.663471 18422 raft_consensus.cc:2726] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:30.642654 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:30.651365 18485 raft_consensus.cc:2781] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election won for term 1
W20260807 03:38:30.665369 18249 leader_election.cc:341] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:30.665762 18485 raft_consensus.cc:686] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:30.666705 18485 consensus_queue.cc:237] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:30.643715 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 83447.6us
I20260807 03:38:30.680979 18366 tablet_bootstrap.cc:654] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:30.691864 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:30.682240 18465 leader_election.cc:288] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
W20260807 03:38:30.701776 18249 leader_election.cc:341] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057): Illegal state: must be running to vote when last-logged opid is not known
W20260807 03:38:30.721365 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 111335us
I20260807 03:38:30.723754 18476 raft_consensus.cc:482] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:30.724251 18476 raft_consensus.cc:504] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:30.726150 18476 leader_election.cc:288] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:30.722982 18471 consensus_queue.cc:1035] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
W20260807 03:38:30.721429 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 156172us
I20260807 03:38:30.738256 18365 tablet_bootstrap.cc:492] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:30.738765 18365 ts_tablet_manager.cc:1397] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.673s user 0.044s sys 0.087s
I20260807 03:38:30.744902 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:30.746230 18365 raft_consensus.cc:348] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:30.746546 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:30.746907 18365 raft_consensus.cc:374] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:30.747186 18365 raft_consensus.cc:729] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
W20260807 03:38:30.747649 18248 leader_election.cc:341] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:30.748116 18248 leader_election.cc:302] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:30.749156 18483 raft_consensus.cc:2726] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260807 03:38:30.750197 18248 leader_election.cc:341] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:30.750638 18248 leader_election.cc:302] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:30.751294 18191 raft_consensus.cc:2454] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:30.751727 18483 raft_consensus.cc:2726] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:30.747797 18365 consensus_queue.cc:260] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:30.766747 18365 ts_tablet_manager.cc:1428] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.028s user 0.005s sys 0.001s
I20260807 03:38:30.767421 18365 tablet_bootstrap.cc:492] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:30.771495 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ca0223f47b46fd955704eba5259370" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:30.765719 18471 consensus_queue.cc:1035] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260807 03:38:30.775202 18309 raft_consensus.cc:3037] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:30.780681 18191 raft_consensus.cc:3037] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:30.790815 18471 consensus_queue.cc:1035] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 03:38:30.801442 17920 catalog_manager.cc:5495] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 1 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:38:30.793774 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ca0223f47b46fd955704eba5259370" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:30.831957 18365 tablet_bootstrap.cc:654] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:30.833926 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:30.835167 18369 tablet_bootstrap.cc:492] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:30.835611 18369 ts_tablet_manager.cc:1397] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.285s user 0.014s sys 0.019s
I20260807 03:38:30.825309 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:38:30.839507 18140 leader_election.cc:341] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
W20260807 03:38:30.841420 18141 consensus_peers.cc:487] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 -> Peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021): Couldn't send request to peer 66a36e1ac8e945e893668e282d394c49. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260807 03:38:30.848052 18309 raft_consensus.cc:2454] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:30.852316 18369 raft_consensus.cc:348] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:30.853000 18369 raft_consensus.cc:374] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:30.853276 18369 raft_consensus.cc:729] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
W20260807 03:38:30.836556 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 46137us
I20260807 03:38:30.853891 18369 consensus_queue.cc:260] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:30.856163 18141 leader_election.cc:302] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:30.857017 18471 raft_consensus.cc:2781] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:30.857452 18471 raft_consensus.cc:686] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:30.858201 18366 tablet_bootstrap.cc:492] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:30.858623 18366 ts_tablet_manager.cc:1397] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.304s user 0.014s sys 0.036s
I20260807 03:38:30.861936 18369 ts_tablet_manager.cc:1428] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.026s user 0.009s sys 0.010s
I20260807 03:38:30.862936 18369 tablet_bootstrap.cc:492] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:30.861698 18418 consensus_queue.cc:1035] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260807 03:38:30.865458 18027 leader_election.cc:302] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:30.866634 18478 raft_consensus.cc:2726] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:30.858218 18471 consensus_queue.cc:237] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:30.870728 18366 raft_consensus.cc:348] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:30.871227 18366 raft_consensus.cc:374] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:30.871415 18366 raft_consensus.cc:729] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:30.872077 18366 consensus_queue.cc:260] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:30.873977 18366 ts_tablet_manager.cc:1428] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20260807 03:38:30.875698 18366 tablet_bootstrap.cc:492] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:30.887022 18366 tablet_bootstrap.cc:654] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:30.889027 18365 tablet_bootstrap.cc:492] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:30.889410 18365 ts_tablet_manager.cc:1397] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.122s user 0.011s sys 0.011s
I20260807 03:38:30.897619 18365 raft_consensus.cc:348] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:30.898093 18365 raft_consensus.cc:374] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:30.898348 18365 raft_consensus.cc:729] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:30.899899 18365 consensus_queue.cc:260] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:30.901628 18369 tablet_bootstrap.cc:654] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:30.906852 18365 ts_tablet_manager.cc:1428] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.017s user 0.003s sys 0.007s
I20260807 03:38:30.907749 18365 tablet_bootstrap.cc:492] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:30.908296 18471 consensus_queue.cc:1035] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260807 03:38:30.930912 18365 tablet_bootstrap.cc:654] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:30.961894 18369 tablet_bootstrap.cc:492] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:30.962512 18369 ts_tablet_manager.cc:1397] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.100s user 0.019s sys 0.007s
I20260807 03:38:30.963354 18081 raft_consensus.cc:3037] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:30.965236 18369 raft_consensus.cc:348] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:30.965984 18369 raft_consensus.cc:374] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:30.966308 18369 raft_consensus.cc:729] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:30.967123 18369 consensus_queue.cc:260] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:30.968762 18369 ts_tablet_manager.cc:1428] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260807 03:38:30.971879 17920 catalog_manager.cc:5495] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 1 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:38:30.974231 17920 catalog_manager.cc:5495] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 1 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:30.976848 18369 tablet_bootstrap.cc:492] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:30.979403 18366 tablet_bootstrap.cc:492] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:30.979919 18366 ts_tablet_manager.cc:1397] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.105s user 0.014s sys 0.005s
I20260807 03:38:30.982719 18366 raft_consensus.cc:348] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:30.983592 18366 raft_consensus.cc:374] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:30.983908 18366 raft_consensus.cc:729] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:30.984676 18366 consensus_queue.cc:260] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:30.987558 18341 debug-util.cc:398] Leaking SignalData structure 0x7b08001a2980 after lost signal to thread 16157
I20260807 03:38:30.987603 18366 ts_tablet_manager.cc:1428] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
W20260807 03:38:30.988596 18341 debug-util.cc:398] Leaking SignalData structure 0x7b08000df6c0 after lost signal to thread 17976
I20260807 03:38:30.988819 18366 tablet_bootstrap.cc:492] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
W20260807 03:38:30.989657 18341 debug-util.cc:398] Leaking SignalData structure 0x7b08000f9480 after lost signal to thread 18113
W20260807 03:38:30.990610 18341 debug-util.cc:398] Leaking SignalData structure 0x7b08001d66e0 after lost signal to thread 18228
W20260807 03:38:30.998660 18341 debug-util.cc:398] Leaking SignalData structure 0x7b0800057aa0 after lost signal to thread 18346
I20260807 03:38:31.010216 18366 tablet_bootstrap.cc:654] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:31.006479 17915 catalog_manager.cc:5495] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 1 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:38:31.017035 18369 tablet_bootstrap.cc:654] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:31.023705 18365 tablet_bootstrap.cc:492] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:31.024237 18365 ts_tablet_manager.cc:1397] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.117s user 0.013s sys 0.023s
I20260807 03:38:31.026738 18365 raft_consensus.cc:348] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.027501 18365 raft_consensus.cc:374] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:31.027761 18365 raft_consensus.cc:729] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:31.028458 18365 consensus_queue.cc:260] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.030763 18365 ts_tablet_manager.cc:1428] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260807 03:38:31.031605 18365 tablet_bootstrap.cc:492] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:31.036902 18365 tablet_bootstrap.cc:654] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:31.048516 18369 tablet_bootstrap.cc:492] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:31.048960 18369 ts_tablet_manager.cc:1397] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.072s user 0.021s sys 0.009s
I20260807 03:38:31.051448 18369 raft_consensus.cc:348] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.052205 18369 raft_consensus.cc:374] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:31.052492 18369 raft_consensus.cc:729] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:31.053190 18369 consensus_queue.cc:260] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.054344 18366 tablet_bootstrap.cc:492] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:31.054760 18366 ts_tablet_manager.cc:1397] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.066s user 0.012s sys 0.014s
I20260807 03:38:31.055454 18365 tablet_bootstrap.cc:492] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:31.055451 18369 ts_tablet_manager.cc:1428] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260807 03:38:31.056149 18365 ts_tablet_manager.cc:1397] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.025s user 0.013s sys 0.003s
I20260807 03:38:31.056571 18369 tablet_bootstrap.cc:492] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:31.058928 18485 raft_consensus.cc:482] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:31.058920 18365 raft_consensus.cc:348] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.059461 18485 raft_consensus.cc:504] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.060012 18365 raft_consensus.cc:374] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260807 03:38:31.060132 18248 consensus_peers.cc:487] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 -> Peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Couldn't send request to peer d9716e2bf06442f5b3b0e81087c53c4d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260807 03:38:31.061453 18485 leader_election.cc:288] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:31.061800 18366 raft_consensus.cc:348] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.062433 18366 raft_consensus.cc:374] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:31.062681 18366 raft_consensus.cc:729] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:31.063418 18366 consensus_queue.cc:260] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.066573 18369 tablet_bootstrap.cc:654] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:31.068333 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:31.068333 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:31.069917 18365 raft_consensus.cc:729] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:31.070081 18191 raft_consensus.cc:2454] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 0.
I20260807 03:38:31.070672 18365 consensus_queue.cc:260] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.075014 18366 ts_tablet_manager.cc:1428] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.020s user 0.006s sys 0.001s
I20260807 03:38:31.071357 18249 leader_election.cc:302] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:31.076624 18485 raft_consensus.cc:2781] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:31.076948 18485 raft_consensus.cc:482] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:31.077292 18485 raft_consensus.cc:3037] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:31.080324 18365 ts_tablet_manager.cc:1428] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.024s user 0.006s sys 0.000s
W20260807 03:38:31.080407 18248 leader_election.cc:341] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:31.081370 18365 tablet_bootstrap.cc:492] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:31.086472 18485 raft_consensus.cc:504] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.091368 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:31.088600 18366 tablet_bootstrap.cc:492] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
W20260807 03:38:31.092653 18248 leader_election.cc:341] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:31.095009 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:31.095583 18191 raft_consensus.cc:3037] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:31.095819 18365 tablet_bootstrap.cc:654] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:31.102885 18191 raft_consensus.cc:2454] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:31.104283 18249 leader_election.cc:302] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:31.105070 18465 raft_consensus.cc:2781] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:31.105511 18465 raft_consensus.cc:686] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:31.106364 18465 consensus_queue.cc:237] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.111155 18366 tablet_bootstrap.cc:654] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:31.111349 18471 raft_consensus.cc:482] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:31.111974 18471 raft_consensus.cc:504] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.114477 18471 leader_election.cc:288] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:31.115471 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:31.115604 18402 raft_consensus.cc:482] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:31.115523 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:31.115932 18485 leader_election.cc:288] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:31.116438 18311 raft_consensus.cc:2454] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 0.
I20260807 03:38:31.116200 18402 raft_consensus.cc:504] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.118155 18141 leader_election.cc:302] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:31.118628 18402 leader_election.cc:288] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:31.119302 18471 raft_consensus.cc:2781] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:31.119808 18471 raft_consensus.cc:482] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:31.120271 18471 raft_consensus.cc:3037] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:31.116200 18081 raft_consensus.cc:2454] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 0.
I20260807 03:38:31.123720 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:38:31.125481 18248 leader_election.cc:341] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:31.126984 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:38:31.128224 18249 leader_election.cc:341] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:31.128702 18249 leader_election.cc:302] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:31.129521 18465 raft_consensus.cc:2726] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:31.130857 18365 tablet_bootstrap.cc:492] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:31.129817 17915 catalog_manager.cc:5495] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 1 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:38:31.132656 18369 tablet_bootstrap.cc:492] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:31.133088 18369 ts_tablet_manager.cc:1397] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.077s user 0.016s sys 0.032s
I20260807 03:38:31.133385 18366 tablet_bootstrap.cc:492] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:31.133803 18366 ts_tablet_manager.cc:1397] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.045s user 0.011s sys 0.009s
I20260807 03:38:31.135463 18369 raft_consensus.cc:348] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.136206 18369 raft_consensus.cc:374] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:31.136549 18369 raft_consensus.cc:729] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:31.136694 18465 consensus_queue.cc:1035] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:31.137157 18369 consensus_queue.cc:260] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.139118 18369 ts_tablet_manager.cc:1428] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20260807 03:38:31.139930 18369 tablet_bootstrap.cc:492] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:31.142021 18365 ts_tablet_manager.cc:1397] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.061s user 0.013s sys 0.002s
I20260807 03:38:31.136673 18366 raft_consensus.cc:348] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.145047 18366 raft_consensus.cc:374] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:31.145362 18366 raft_consensus.cc:729] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:31.148036 18369 tablet_bootstrap.cc:654] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:31.147632 18366 consensus_queue.cc:260] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.128026 18471 raft_consensus.cc:504] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.151623 18366 ts_tablet_manager.cc:1428] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.018s user 0.004s sys 0.004s
I20260807 03:38:31.152791 18366 tablet_bootstrap.cc:492] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:31.155130 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:31.155200 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:31.155418 18465 consensus_queue.cc:1035] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:31.155762 18311 raft_consensus.cc:3037] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:31.156005 18080 raft_consensus.cc:3037] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:31.159976 18369 tablet_bootstrap.cc:492] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:31.160393 18369 ts_tablet_manager.cc:1397] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.021s user 0.016s sys 0.003s
I20260807 03:38:31.162815 18369 raft_consensus.cc:348] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.163486 18369 raft_consensus.cc:374] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:31.163374 18080 raft_consensus.cc:2454] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:31.163777 18369 raft_consensus.cc:729] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:31.163947 18311 raft_consensus.cc:2454] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:31.164587 18140 leader_election.cc:302] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:31.165269 18471 leader_election.cc:288] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:31.144562 18365 raft_consensus.cc:348] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.166347 18422 raft_consensus.cc:2781] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:31.169142 18422 raft_consensus.cc:686] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:31.164577 18369 consensus_queue.cc:260] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.170025 18422 consensus_queue.cc:237] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.172370 18369 ts_tablet_manager.cc:1428] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260807 03:38:31.173585 18369 tablet_bootstrap.cc:492] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:31.177438 18365 raft_consensus.cc:374] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:31.177805 18365 raft_consensus.cc:729] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:31.180431 18369 tablet_bootstrap.cc:654] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:31.179320 18365 consensus_queue.cc:260] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.184639 18365 ts_tablet_manager.cc:1428] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.042s user 0.006s sys 0.000s
I20260807 03:38:31.188825 18365 tablet_bootstrap.cc:492] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:31.190800 18366 tablet_bootstrap.cc:654] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:31.191447 18402 consensus_queue.cc:1035] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 03:38:31.193389 18478 raft_consensus.cc:482] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:31.193852 18478 raft_consensus.cc:504] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.195935 18478 leader_election.cc:288] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:31.198117 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:31.203503 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:31.204430 18311 raft_consensus.cc:2454] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 0.
I20260807 03:38:31.210808 18027 leader_election.cc:302] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:31.211725 18507 raft_consensus.cc:2781] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:31.212139 18507 raft_consensus.cc:482] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:31.212518 18507 raft_consensus.cc:3037] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:31.201563 18193 raft_consensus.cc:2454] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 0.
I20260807 03:38:31.218321 18365 tablet_bootstrap.cc:654] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:31.235170 18507 raft_consensus.cc:504] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.237203 18507 leader_election.cc:288] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:31.237577 18422 raft_consensus.cc:482] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:31.238206 18422 raft_consensus.cc:504] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.238770 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:31.239401 18193 raft_consensus.cc:3037] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:31.241623 18422 leader_election.cc:288] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:31.242539 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:38:31.244227 18141 leader_election.cc:341] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:31.245750 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:38:31.248068 18140 leader_election.cc:341] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:31.248644 18140 leader_election.cc:302] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:31.250178 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:31.250953 18311 raft_consensus.cc:3037] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:31.250649 17920 catalog_manager.cc:5495] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 1 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:31.260905 18193 raft_consensus.cc:2454] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:38:31.256513 18422 raft_consensus.cc:2726] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:31.262461 18027 leader_election.cc:302] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:31.263868 18478 raft_consensus.cc:2781] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:31.264340 18478 raft_consensus.cc:686] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:38:31.265170 18478 consensus_queue.cc:237] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.269163 18311 raft_consensus.cc:2454] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:38:31.285362 18369 tablet_bootstrap.cc:492] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:31.285842 18369 ts_tablet_manager.cc:1397] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.112s user 0.007s sys 0.016s
I20260807 03:38:31.290572 18369 raft_consensus.cc:348] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.291402 18369 raft_consensus.cc:374] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:31.291666 18369 raft_consensus.cc:729] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:31.292317 18369 consensus_queue.cc:260] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.294291 18369 ts_tablet_manager.cc:1428] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.008s user 0.001s sys 0.005s
I20260807 03:38:31.295040 18369 tablet_bootstrap.cc:492] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:31.299218 18365 tablet_bootstrap.cc:492] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:31.298264 17912 catalog_manager.cc:5495] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 0 to 1, leader changed from <none> to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 1 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:31.299685 18365 ts_tablet_manager.cc:1397] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.111s user 0.025s sys 0.027s
I20260807 03:38:31.300299 18366 tablet_bootstrap.cc:492] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:31.300736 18366 ts_tablet_manager.cc:1397] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.148s user 0.028s sys 0.012s
I20260807 03:38:31.303790 18366 raft_consensus.cc:348] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.304457 18366 raft_consensus.cc:374] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:31.304728 18366 raft_consensus.cc:729] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:31.305311 18366 consensus_queue.cc:260] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.307318 18366 ts_tablet_manager.cc:1428] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260807 03:38:31.308221 18366 tablet_bootstrap.cc:492] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:31.312860 18365 raft_consensus.cc:348] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.313571 18365 raft_consensus.cc:374] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:31.313864 18365 raft_consensus.cc:729] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:31.314488 18365 consensus_queue.cc:260] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.315685 18422 raft_consensus.cc:482] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:31.316179 18422 raft_consensus.cc:504] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.316874 18366 tablet_bootstrap.cc:654] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:31.317974 18365 ts_tablet_manager.cc:1428] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.018s user 0.006s sys 0.001s
I20260807 03:38:31.318878 18365 tablet_bootstrap.cc:492] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:31.319108 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:38:31.320907 18140 leader_election.cc:341] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:31.323515 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:31.324368 18312 raft_consensus.cc:2454] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 0.
I20260807 03:38:31.326030 18141 leader_election.cc:302] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:31.327242 18471 raft_consensus.cc:2781] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:31.327687 18471 raft_consensus.cc:482] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:31.328454 18471 raft_consensus.cc:3037] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:31.334709 18422 leader_election.cc:288] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:31.335630 18366 tablet_bootstrap.cc:492] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:31.336045 18366 ts_tablet_manager.cc:1397] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.019s
I20260807 03:38:31.335981 18465 raft_consensus.cc:482] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:31.336640 18465 raft_consensus.cc:504] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.338717 18465 leader_election.cc:288] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:31.338765 18366 raft_consensus.cc:348] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.339452 18366 raft_consensus.cc:374] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:31.339757 18366 raft_consensus.cc:729] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:31.336155 18365 tablet_bootstrap.cc:654] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:31.340361 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:31.341212 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:38:31.341835 18249 leader_election.cc:341] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057): Illegal state: must be running to vote when last-logged opid is not known
W20260807 03:38:31.342489 18248 leader_election.cc:341] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:31.342231 18366 consensus_queue.cc:260] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.342923 18248 leader_election.cc:302] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:31.343645 18465 raft_consensus.cc:2726] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:31.344429 18366 ts_tablet_manager.cc:1428] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260807 03:38:31.345654 18366 tablet_bootstrap.cc:492] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:31.345844 18471 raft_consensus.cc:504] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.348642 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:31.349949 18369 tablet_bootstrap.cc:654] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
W20260807 03:38:31.350140 18140 leader_election.cc:341] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:31.351852 18471 leader_election.cc:288] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:31.353300 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:31.353909 18312 raft_consensus.cc:3037] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:31.355494 18366 tablet_bootstrap.cc:654] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:31.365145 18369 tablet_bootstrap.cc:492] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:31.365628 18369 ts_tablet_manager.cc:1397] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.071s user 0.008s sys 0.004s
I20260807 03:38:31.365935 18576 raft_consensus.cc:482] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:31.366955 18576 raft_consensus.cc:504] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.370250 18576 leader_election.cc:288] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:31.372494 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:31.373171 18191 raft_consensus.cc:2454] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 0.
I20260807 03:38:31.373781 18312 raft_consensus.cc:2454] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:31.374505 18027 leader_election.cc:302] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:31.377637 18365 tablet_bootstrap.cc:492] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:31.378177 18365 ts_tablet_manager.cc:1397] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.059s user 0.012s sys 0.015s
I20260807 03:38:31.378131 18141 leader_election.cc:302] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:31.378492 18576 raft_consensus.cc:2781] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:31.378842 18576 raft_consensus.cc:482] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:31.379140 18576 raft_consensus.cc:3037] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:31.379398 18422 raft_consensus.cc:2781] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:31.379910 18422 raft_consensus.cc:686] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:31.380756 18422 consensus_queue.cc:237] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.381134 18365 raft_consensus.cc:348] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.382074 18365 raft_consensus.cc:374] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:31.378281 18471 consensus_queue.cc:1035] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260807 03:38:31.382422 18365 raft_consensus.cc:729] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:31.383144 18365 consensus_queue.cc:260] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.374688 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:31.385277 18365 ts_tablet_manager.cc:1428] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260807 03:38:31.386219 18365 tablet_bootstrap.cc:492] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:31.385107 18311 raft_consensus.cc:2454] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 0.
I20260807 03:38:31.390991 18576 raft_consensus.cc:504] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.393697 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:31.394325 18365 tablet_bootstrap.cc:654] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:31.394258 18191 raft_consensus.cc:3037] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:31.376542 18369 raft_consensus.cc:348] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.401770 18369 raft_consensus.cc:374] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:31.402456 18369 raft_consensus.cc:729] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:31.403461 18369 consensus_queue.cc:260] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.406668 18191 raft_consensus.cc:2454] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:38:31.412053 18366 tablet_bootstrap.cc:492] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:31.412220 18027 leader_election.cc:302] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:31.412679 18366 ts_tablet_manager.cc:1397] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.067s user 0.012s sys 0.002s
I20260807 03:38:31.413296 18574 raft_consensus.cc:2781] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:31.413834 18574 raft_consensus.cc:686] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:38:31.414646 18574 consensus_queue.cc:237] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.415804 18366 raft_consensus.cc:348] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.416709 18366 raft_consensus.cc:374] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:31.417034 18366 raft_consensus.cc:729] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:31.417729 18366 consensus_queue.cc:260] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.420883 18366 ts_tablet_manager.cc:1428] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260807 03:38:31.422058 18366 tablet_bootstrap.cc:492] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:31.426656 18365 tablet_bootstrap.cc:492] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:31.426592 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:31.427228 18365 ts_tablet_manager.cc:1397] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.041s user 0.018s sys 0.020s
I20260807 03:38:31.427309 18311 raft_consensus.cc:3037] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:31.429152 18366 tablet_bootstrap.cc:654] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:31.419924 18369 ts_tablet_manager.cc:1428] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.054s user 0.008s sys 0.000s
I20260807 03:38:31.429745 18365 raft_consensus.cc:348] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.430485 18365 raft_consensus.cc:374] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:31.430739 18365 raft_consensus.cc:729] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:31.431507 18365 consensus_queue.cc:260] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.432818 18369 tablet_bootstrap.cc:492] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:31.434510 18365 ts_tablet_manager.cc:1428] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260807 03:38:31.435148 18311 raft_consensus.cc:2454] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:38:31.436324 18576 leader_election.cc:288] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:31.441042 18365 tablet_bootstrap.cc:492] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:31.443455 18081 raft_consensus.cc:3037] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:31.443291 17912 catalog_manager.cc:5495] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 1 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:31.450234 18574 raft_consensus.cc:482] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:31.450752 18574 raft_consensus.cc:504] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.449841 17915 catalog_manager.cc:5495] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 0 to 1, leader changed from <none> to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 1 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:31.453011 18574 leader_election.cc:288] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:31.455950 18365 tablet_bootstrap.cc:654] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:31.462378 18366 tablet_bootstrap.cc:492] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:31.462771 18366 ts_tablet_manager.cc:1397] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.041s user 0.008s sys 0.002s
I20260807 03:38:31.463459 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:31.460320 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:31.462088 18422 raft_consensus.cc:482] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:31.468829 18422 raft_consensus.cc:504] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.472209 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:38:31.473505 18140 leader_election.cc:341] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:31.476012 18366 raft_consensus.cc:348] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.476966 18366 raft_consensus.cc:374] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:31.477281 18366 raft_consensus.cc:729] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:31.478003 18366 consensus_queue.cc:260] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.479085 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:38:31.480541 18141 leader_election.cc:341] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:31.481195 18141 leader_election.cc:302] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:31.481400 18366 ts_tablet_manager.cc:1428] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.018s user 0.009s sys 0.000s
I20260807 03:38:31.482249 18469 raft_consensus.cc:2726] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:31.482642 18366 tablet_bootstrap.cc:492] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:31.482929 18422 leader_election.cc:288] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:31.488487 18027 leader_election.cc:302] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:31.489326 18080 raft_consensus.cc:3037] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:31.489706 18369 tablet_bootstrap.cc:654] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:31.497040 18469 consensus_queue.cc:1035] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:31.503571 18365 tablet_bootstrap.cc:492] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:31.504112 18365 ts_tablet_manager.cc:1397] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.063s user 0.019s sys 0.014s
I20260807 03:38:31.506747 18365 raft_consensus.cc:348] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.507484 18365 raft_consensus.cc:374] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:31.507731 18365 raft_consensus.cc:729] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:31.510208 18366 tablet_bootstrap.cc:654] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:31.514595 18365 consensus_queue.cc:260] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.517109 18365 ts_tablet_manager.cc:1428] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260807 03:38:31.517951 18365 tablet_bootstrap.cc:492] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:31.523061 18081 raft_consensus.cc:3037] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:31.530524 18365 tablet_bootstrap.cc:654] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:31.544817 18574 raft_consensus.cc:2726] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:31.553395 18369 tablet_bootstrap.cc:492] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:31.553853 18369 ts_tablet_manager.cc:1397] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.121s user 0.027s sys 0.031s
W20260807 03:38:31.559155 18003 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 03:38:31.561218 18366 tablet_bootstrap.cc:492] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:31.561699 18366 ts_tablet_manager.cc:1397] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.079s user 0.012s sys 0.019s
I20260807 03:38:31.564625 18365 tablet_bootstrap.cc:492] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:31.565307 18366 raft_consensus.cc:348] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.566170 18366 raft_consensus.cc:374] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:31.566618 18366 raft_consensus.cc:729] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:31.566398 18369 raft_consensus.cc:348] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.567265 18369 raft_consensus.cc:374] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:31.566135 18365 ts_tablet_manager.cc:1397] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.048s user 0.011s sys 0.005s
I20260807 03:38:31.567370 18469 consensus_queue.cc:1035] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 03:38:31.567348 18366 consensus_queue.cc:260] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.570236 18366 ts_tablet_manager.cc:1428] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.008s user 0.004s sys 0.002s
I20260807 03:38:31.570166 18365 raft_consensus.cc:348] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.570899 18365 raft_consensus.cc:374] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:31.571125 18366 tablet_bootstrap.cc:492] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:31.571197 18365 raft_consensus.cc:729] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:31.572185 18365 consensus_queue.cc:260] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.574847 18369 raft_consensus.cc:729] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:31.575644 18369 consensus_queue.cc:260] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.574651 18365 ts_tablet_manager.cc:1428] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20260807 03:38:31.577229 18365 tablet_bootstrap.cc:492] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:31.577630 18469 raft_consensus.cc:482] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:31.577865 18369 ts_tablet_manager.cc:1428] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.024s user 0.008s sys 0.000s
I20260807 03:38:31.578146 18469 raft_consensus.cc:504] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.578850 18369 tablet_bootstrap.cc:492] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:31.582929 18365 tablet_bootstrap.cc:654] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:31.585078 18369 tablet_bootstrap.cc:654] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:31.587782 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:31.588616 18311 raft_consensus.cc:2454] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 0.
I20260807 03:38:31.590411 18141 leader_election.cc:302] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:31.591909 18471 raft_consensus.cc:2781] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:31.592486 18471 raft_consensus.cc:482] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:31.581667 18469 leader_election.cc:288] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:31.592943 18471 raft_consensus.cc:3037] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:31.582707 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:31.596710 18082 raft_consensus.cc:2454] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 0.
I20260807 03:38:31.604820 18576 raft_consensus.cc:482] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:31.605553 18576 raft_consensus.cc:504] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.616312 18576 leader_election.cc:288] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:31.616940 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:31.617324 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:31.619688 18027 leader_election.cc:302] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:31.622328 18422 consensus_queue.cc:1035] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 03:38:31.632659 18469 raft_consensus.cc:482] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:31.633309 18469 raft_consensus.cc:504] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.636878 18469 leader_election.cc:288] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:31.637200 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:31.638931 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:31.650228 18082 raft_consensus.cc:2454] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 0.
W20260807 03:38:31.651499 18141 leader_election.cc:341] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:31.640470 18576 raft_consensus.cc:2726] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:31.645679 18471 raft_consensus.cc:504] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.654767 18140 leader_election.cc:302] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:31.657214 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:31.657413 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:31.657931 18312 raft_consensus.cc:3037] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:31.658231 18082 raft_consensus.cc:3037] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:31.664290 18469 raft_consensus.cc:2781] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:31.665199 18469 raft_consensus.cc:482] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:31.666568 18469 raft_consensus.cc:3037] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:31.667793 18082 raft_consensus.cc:2454] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:31.677127 18140 leader_election.cc:302] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:31.648442 18366 tablet_bootstrap.cc:654] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
W20260807 03:38:31.682279 18248 consensus_peers.cc:487] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 -> Peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Couldn't send request to peer d9716e2bf06442f5b3b0e81087c53c4d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260807 03:38:31.683391 18422 raft_consensus.cc:2781] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:31.694422 18471 leader_election.cc:288] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:31.704730 18465 raft_consensus.cc:482] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:31.708505 18465 raft_consensus.cc:504] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.706838 18312 raft_consensus.cc:2454] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:31.709676 18369 tablet_bootstrap.cc:492] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:31.710093 18369 ts_tablet_manager.cc:1397] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.131s user 0.055s sys 0.063s
I20260807 03:38:31.710404 18465 leader_election.cc:288] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:31.710868 18465 raft_consensus.cc:482] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:31.713096 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:31.711279 18465 raft_consensus.cc:504] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.715034 18469 raft_consensus.cc:504] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.717317 18485 consensus_queue.cc:1035] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:31.717931 18469 leader_election.cc:288] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:31.715629 18369 raft_consensus.cc:348] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.718467 18369 raft_consensus.cc:374] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:31.718788 18369 raft_consensus.cc:729] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:31.717262 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:31.719074 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:31.719240 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:31.771438 18619 raft_consensus.cc:482] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:31.719408 18369 consensus_queue.cc:260] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.736745 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:31.783228 18365 tablet_bootstrap.cc:492] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:31.783378 18465 leader_election.cc:288] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:31.783605 18193 raft_consensus.cc:2454] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 0.
I20260807 03:38:31.782611 18619 raft_consensus.cc:504] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.784356 18365 ts_tablet_manager.cc:1397] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.207s user 0.034s sys 0.111s
I20260807 03:38:31.784816 18366 tablet_bootstrap.cc:492] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:31.785260 18366 ts_tablet_manager.cc:1397] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.214s user 0.016s sys 0.024s
I20260807 03:38:31.786334 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:31.734440 18248 leader_election.cc:302] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:31.786813 18365 raft_consensus.cc:348] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.787106 18249 leader_election.cc:302] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:31.787441 18365 raft_consensus.cc:374] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:31.787788 18365 raft_consensus.cc:729] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
W20260807 03:38:31.787848 18140 leader_election.cc:341] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:31.788432 18402 raft_consensus.cc:2726] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:31.789055 18469 raft_consensus.cc:482] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:31.789212 18080 raft_consensus.cc:3037] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:31.789148 18402 raft_consensus.cc:2781] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:31.789491 18619 leader_election.cc:288] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:31.789894 18402 raft_consensus.cc:482] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:31.789530 18469 raft_consensus.cc:504] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.790277 18402 raft_consensus.cc:3037] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:31.704689 18422 raft_consensus.cc:686] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:31.791705 18469 leader_election.cc:288] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:31.723786 18081 raft_consensus.cc:2454] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 0.
I20260807 03:38:31.788586 18365 consensus_queue.cc:260] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:31.793561 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 59236.4us
I20260807 03:38:31.793570 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:38:31.795262 18140 leader_election.cc:341] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:31.797116 18365 ts_tablet_manager.cc:1428] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.012s user 0.004s sys 0.002s
I20260807 03:38:31.797933 18365 tablet_bootstrap.cc:492] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:31.798069 18402 raft_consensus.cc:504] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.798696 18080 raft_consensus.cc:2454] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:31.799839 18140 leader_election.cc:302] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:31.800978 18469 raft_consensus.cc:2781] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:31.801316 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:31.802017 18190 raft_consensus.cc:3037] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:31.801497 18469 raft_consensus.cc:686] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:31.802297 18576 consensus_queue.cc:1035] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:31.803153 18469 consensus_queue.cc:237] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.804908 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:31.805119 18366 raft_consensus.cc:348] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.805470 18311 raft_consensus.cc:3037] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:31.805723 18366 raft_consensus.cc:374] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:31.805996 18366 raft_consensus.cc:729] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:31.807879 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:31.809894 18082 raft_consensus.cc:3037] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:31.811051 18366 consensus_queue.cc:260] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.811968 18311 raft_consensus.cc:2454] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:31.814414 18369 ts_tablet_manager.cc:1428] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.104s user 0.010s sys 0.006s
I20260807 03:38:31.815325 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:31.815927 18311 raft_consensus.cc:2454] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 0.
I20260807 03:38:31.818598 18369 tablet_bootstrap.cc:492] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:31.821091 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:31.822494 18366 ts_tablet_manager.cc:1428] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.037s user 0.003s sys 0.002s
I20260807 03:38:31.807996 18422 consensus_queue.cc:237] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.825456 18312 raft_consensus.cc:3037] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:31.825803 18369 tablet_bootstrap.cc:654] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:31.834022 18369 tablet_bootstrap.cc:492] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:31.834745 18369 ts_tablet_manager.cc:1397] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.001s
I20260807 03:38:31.835453 18141 leader_election.cc:302] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:31.835753 18365 tablet_bootstrap.cc:654] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:31.836565 18619 raft_consensus.cc:2781] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:31.836869 18619 raft_consensus.cc:482] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:31.837186 18619 raft_consensus.cc:3037] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
W20260807 03:38:31.839010 18141 leader_election.cc:341] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:31.837191 18369 raft_consensus.cc:348] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.839550 18141 leader_election.cc:302] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:31.840540 18366 tablet_bootstrap.cc:492] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:31.840770 18471 raft_consensus.cc:2726] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:31.839699 18369 raft_consensus.cc:374] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:31.842815 18369 raft_consensus.cc:729] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:31.843487 18369 consensus_queue.cc:260] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.844274 18402 leader_election.cc:288] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:31.845571 18369 ts_tablet_manager.cc:1428] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20260807 03:38:31.849534 18369 tablet_bootstrap.cc:492] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:31.853971 18082 raft_consensus.cc:2454] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:31.854661 18576 consensus_queue.cc:1035] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:31.855250 18248 leader_election.cc:302] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:31.856208 18402 raft_consensus.cc:2781] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:31.856226 17920 catalog_manager.cc:5495] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 1 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:31.863709 18369 tablet_bootstrap.cc:654] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:31.876618 18190 raft_consensus.cc:2454] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:31.870492 18402 raft_consensus.cc:686] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:31.888798 18632 raft_consensus.cc:482] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:31.903849 18478 consensus_queue.cc:1035] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:31.904906 18365 tablet_bootstrap.cc:492] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:31.905232 18402 consensus_queue.cc:237] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.905365 18365 ts_tablet_manager.cc:1397] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.108s user 0.031s sys 0.019s
I20260807 03:38:31.889469 18632 raft_consensus.cc:504] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:31.919662 18365 raft_consensus.cc:348] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.015787 18365 raft_consensus.cc:374] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:32.016065 18366 tablet_bootstrap.cc:654] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:32.016155 18365 raft_consensus.cc:729] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:32.016870 18365 consensus_queue.cc:260] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.018388 18365 ts_tablet_manager.cc:1428] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.110s user 0.003s sys 0.031s
I20260807 03:38:32.019295 18365 tablet_bootstrap.cc:492] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:32.020803 18369 tablet_bootstrap.cc:492] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:32.021219 18369 ts_tablet_manager.cc:1397] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.175s user 0.028s sys 0.009s
I20260807 03:38:31.948446 18465 raft_consensus.cc:482] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:32.023334 18465 raft_consensus.cc:504] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.023661 18369 raft_consensus.cc:348] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.026486 18369 raft_consensus.cc:374] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:32.026810 18369 raft_consensus.cc:729] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:32.027164 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:32.027556 18369 consensus_queue.cc:260] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:32.029013 18249 leader_election.cc:341] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:32.029357 18369 ts_tablet_manager.cc:1428] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260807 03:38:32.030239 18369 tablet_bootstrap.cc:492] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:31.958590 18632 leader_election.cc:288] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:32.030812 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:32.031435 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:32.032652 18465 leader_election.cc:288] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:32.036324 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:32.037909 17920 catalog_manager.cc:5495] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 1 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:32.039508 18365 tablet_bootstrap.cc:654] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:32.039417 18469 consensus_queue.cc:1035] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 03:38:32.040988 18141 leader_election.cc:341] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021): Illegal state: must be running to vote when last-logged opid is not known
W20260807 03:38:32.043072 18140 leader_election.cc:341] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:32.043568 18140 leader_election.cc:302] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:32.068868 18469 raft_consensus.cc:2726] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260807 03:38:32.048372 18248 leader_election.cc:341] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:32.073323 18248 leader_election.cc:302] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:31.922044 18623 raft_consensus.cc:482] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:32.050351 18369 tablet_bootstrap.cc:654] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:32.074909 18623 raft_consensus.cc:504] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.077018 18623 leader_election.cc:288] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:32.077705 18623 raft_consensus.cc:2726] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:32.078426 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:32.079739 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:38:32.080265 18249 leader_election.cc:341] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:32.053229 18619 raft_consensus.cc:504] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.055382 18478 raft_consensus.cc:482] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 03:38:32.082096 18248 leader_election.cc:341] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:32.081908 18478 raft_consensus.cc:504] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.083799 18478 leader_election.cc:288] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:32.066821 18576 consensus_queue.cc:1035] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 03:38:32.084764 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:32.085089 18248 leader_election.cc:302] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:32.085333 18312 raft_consensus.cc:3037] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:32.092960 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:32.094662 18638 raft_consensus.cc:2726] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:32.098582 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:32.100885 18312 raft_consensus.cc:2454] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:32.101825 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:32.101996 18141 leader_election.cc:302] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:32.102905 18469 raft_consensus.cc:2781] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:32.103446 18469 raft_consensus.cc:686] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:32.104430 18469 consensus_queue.cc:237] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.112587 18619 leader_election.cc:288] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:32.131642 18369 tablet_bootstrap.cc:492] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:32.132131 18369 ts_tablet_manager.cc:1397] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.102s user 0.011s sys 0.008s
I20260807 03:38:32.134239 18080 raft_consensus.cc:3037] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:32.135332 18369 raft_consensus.cc:348] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.136006 18369 raft_consensus.cc:374] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:32.136278 18369 raft_consensus.cc:729] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:32.136876 18369 consensus_queue.cc:260] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.138780 18369 ts_tablet_manager.cc:1428] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260807 03:38:32.139588 18369 tablet_bootstrap.cc:492] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:32.148262 17920 catalog_manager.cc:5495] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 1 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:38:32.154556 18369 tablet_bootstrap.cc:654] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:32.160219 18027 leader_election.cc:302] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:32.167935 18365 tablet_bootstrap.cc:492] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:32.168541 18365 ts_tablet_manager.cc:1397] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.149s user 0.064s sys 0.066s
W20260807 03:38:32.157299 18140 consensus_peers.cc:487] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 -> Peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Couldn't send request to peer d9716e2bf06442f5b3b0e81087c53c4d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260807 03:38:32.173995 18140 leader_election.cc:341] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:32.175196 18366 tablet_bootstrap.cc:492] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:32.175595 18366 ts_tablet_manager.cc:1397] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.335s user 0.049s sys 0.120s
I20260807 03:38:32.176606 18369 tablet_bootstrap.cc:492] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:32.176954 18369 ts_tablet_manager.cc:1397] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.038s user 0.010s sys 0.007s
I20260807 03:38:32.181856 18365 raft_consensus.cc:348] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.183374 18365 raft_consensus.cc:374] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:32.183640 18365 raft_consensus.cc:729] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:32.184281 18365 consensus_queue.cc:260] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.188815 18365 ts_tablet_manager.cc:1428] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20260807 03:38:32.189651 18365 tablet_bootstrap.cc:492] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:32.196364 18366 raft_consensus.cc:348] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.205616 18366 raft_consensus.cc:374] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:32.210731 18366 raft_consensus.cc:729] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:32.220670 18366 consensus_queue.cc:260] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.224732 18366 ts_tablet_manager.cc:1428] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.049s user 0.008s sys 0.000s
I20260807 03:38:32.225005 18365 tablet_bootstrap.cc:654] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:32.229892 18366 tablet_bootstrap.cc:492] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:32.230239 18478 raft_consensus.cc:2726] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:32.233896 17920 catalog_manager.cc:5495] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 1 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:32.235544 18366 tablet_bootstrap.cc:654] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:32.225639 18369 raft_consensus.cc:348] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.237241 18369 raft_consensus.cc:374] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:32.237476 18369 raft_consensus.cc:729] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:32.236498 18619 consensus_queue.cc:1035] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 03:38:32.237977 18632 consensus_queue.cc:1035] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:32.238051 18369 consensus_queue.cc:260] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.240269 18369 ts_tablet_manager.cc:1428] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.063s user 0.007s sys 0.007s
I20260807 03:38:32.241216 18369 tablet_bootstrap.cc:492] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:32.262727 18366 tablet_bootstrap.cc:492] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:32.263998 18366 ts_tablet_manager.cc:1397] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.014s
I20260807 03:38:32.264698 18369 tablet_bootstrap.cc:654] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:32.272053 18366 raft_consensus.cc:348] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.273605 18365 tablet_bootstrap.cc:492] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:32.274010 18365 ts_tablet_manager.cc:1397] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.085s user 0.023s sys 0.004s
I20260807 03:38:32.272965 18366 raft_consensus.cc:374] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:32.274541 18366 raft_consensus.cc:729] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:32.275134 18366 consensus_queue.cc:260] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.276433 18469 consensus_queue.cc:1035] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:32.277015 18366 ts_tablet_manager.cc:1428] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20260807 03:38:32.277794 18366 tablet_bootstrap.cc:492] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:32.283645 18365 raft_consensus.cc:348] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.284366 18365 raft_consensus.cc:374] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:32.284652 18365 raft_consensus.cc:729] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:32.285279 18365 consensus_queue.cc:260] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.287353 18365 ts_tablet_manager.cc:1428] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.013s user 0.004s sys 0.000s
I20260807 03:38:32.288496 18365 tablet_bootstrap.cc:492] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:32.293447 18366 tablet_bootstrap.cc:654] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:32.309585 18369 tablet_bootstrap.cc:492] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:32.310067 18369 ts_tablet_manager.cc:1397] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.069s user 0.010s sys 0.001s
I20260807 03:38:32.294778 18365 tablet_bootstrap.cc:654] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:32.312608 18369 raft_consensus.cc:348] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.313472 18369 raft_consensus.cc:374] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:32.313923 18369 raft_consensus.cc:729] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:32.322207 18619 consensus_queue.cc:1035] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:32.314600 18369 consensus_queue.cc:260] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.324043 18639 raft_consensus.cc:482] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:32.324522 18639 raft_consensus.cc:504] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.328809 18639 leader_election.cc:288] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:32.329442 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:32.330055 18369 ts_tablet_manager.cc:1428] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.020s user 0.005s sys 0.000s
W20260807 03:38:32.331069 18249 leader_election.cc:341] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:32.331167 18369 tablet_bootstrap.cc:492] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:32.332306 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:38:32.333379 18248 leader_election.cc:341] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:32.333887 18248 leader_election.cc:302] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:32.334997 18639 raft_consensus.cc:2726] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:32.343243 18365 tablet_bootstrap.cc:492] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:32.343676 18365 ts_tablet_manager.cc:1397] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.056s user 0.019s sys 0.011s
I20260807 03:38:32.346158 18366 tablet_bootstrap.cc:492] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:32.346163 18365 raft_consensus.cc:348] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.346738 18366 ts_tablet_manager.cc:1397] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.069s user 0.013s sys 0.000s
I20260807 03:38:32.346966 18365 raft_consensus.cc:374] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:32.347226 18365 raft_consensus.cc:729] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:32.347813 18365 consensus_queue.cc:260] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.349220 18366 raft_consensus.cc:348] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.349893 18366 raft_consensus.cc:374] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:32.350173 18366 raft_consensus.cc:729] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:32.350838 18366 consensus_queue.cc:260] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.352861 18366 ts_tablet_manager.cc:1428] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260807 03:38:32.356781 18366 tablet_bootstrap.cc:492] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:32.353401 18369 tablet_bootstrap.cc:654] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:32.353820 18365 ts_tablet_manager.cc:1428] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260807 03:38:32.362562 18365 tablet_bootstrap.cc:492] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:32.368799 18365 tablet_bootstrap.cc:654] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:32.374248 18366 tablet_bootstrap.cc:654] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:32.385260 18369 tablet_bootstrap.cc:492] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:32.385789 18369 ts_tablet_manager.cc:1397] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.055s user 0.010s sys 0.001s
I20260807 03:38:32.390069 18369 raft_consensus.cc:348] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.391960 18369 raft_consensus.cc:374] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:32.393077 18369 raft_consensus.cc:729] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:32.394536 18369 consensus_queue.cc:260] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.397333 18369 ts_tablet_manager.cc:1428] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20260807 03:38:32.399556 18469 raft_consensus.cc:482] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:32.400120 18469 raft_consensus.cc:504] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.402161 18469 leader_election.cc:288] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:32.404175 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:38:32.405678 18140 leader_election.cc:341] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:32.411280 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:32.411922 18311 raft_consensus.cc:2454] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 0.
I20260807 03:38:32.413162 18141 leader_election.cc:302] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:32.414340 18469 raft_consensus.cc:2781] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:32.414657 18469 raft_consensus.cc:482] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:32.415035 18469 raft_consensus.cc:3037] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:32.416992 18369 tablet_bootstrap.cc:492] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:32.428334 18369 tablet_bootstrap.cc:654] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:32.428617 18469 raft_consensus.cc:504] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.431442 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
W20260807 03:38:32.436305 18140 leader_election.cc:341] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:32.440567 18619 raft_consensus.cc:482] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:32.441162 18619 raft_consensus.cc:504] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.443742 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:32.444398 18311 raft_consensus.cc:3037] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:32.444953 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:32.445888 18369 tablet_bootstrap.cc:492] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:32.446302 18369 ts_tablet_manager.cc:1397] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.029s user 0.007s sys 0.003s
W20260807 03:38:32.446537 18141 leader_election.cc:341] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:32.448688 18369 raft_consensus.cc:348] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.449383 18369 raft_consensus.cc:374] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:32.449666 18369 raft_consensus.cc:729] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:32.451133 18311 raft_consensus.cc:2454] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:32.450908 18369 consensus_queue.cc:260] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.452363 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:32.452647 18141 leader_election.cc:302] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:32.452962 18369 ts_tablet_manager.cc:1428] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260807 03:38:32.453555 18471 raft_consensus.cc:2781] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:32.454035 18469 leader_election.cc:288] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:32.454159 18471 raft_consensus.cc:686] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:32.455369 18471 consensus_queue.cc:237] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.458997 18369 tablet_bootstrap.cc:492] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:32.459710 18366 tablet_bootstrap.cc:492] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:32.459656 18619 leader_election.cc:288] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:32.460230 18366 ts_tablet_manager.cc:1397] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.104s user 0.040s sys 0.049s
W20260807 03:38:32.460480 18140 leader_election.cc:341] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:32.461042 18140 leader_election.cc:302] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:32.462276 18422 raft_consensus.cc:2726] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:32.462774 18366 raft_consensus.cc:348] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.463425 18366 raft_consensus.cc:374] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:32.463747 18366 raft_consensus.cc:729] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:32.464829 18369 tablet_bootstrap.cc:654] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:32.464555 18366 consensus_queue.cc:260] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.466172 18365 tablet_bootstrap.cc:492] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:32.466571 18365 ts_tablet_manager.cc:1397] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.104s user 0.026s sys 0.018s
I20260807 03:38:32.466995 18366 ts_tablet_manager.cc:1428] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260807 03:38:32.467779 18366 tablet_bootstrap.cc:492] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:32.474133 18366 tablet_bootstrap.cc:654] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:32.475805 18365 raft_consensus.cc:348] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.477914 18365 raft_consensus.cc:374] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:32.479140 18365 raft_consensus.cc:729] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:32.480698 18365 consensus_queue.cc:260] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.485767 18639 raft_consensus.cc:482] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:32.486447 18365 ts_tablet_manager.cc:1428] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.020s user 0.008s sys 0.000s
I20260807 03:38:32.486233 18639 raft_consensus.cc:504] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.488427 18365 tablet_bootstrap.cc:492] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:32.489084 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:38:32.490612 18248 leader_election.cc:341] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:32.491957 18639 leader_election.cc:288] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:32.492237 17912 catalog_manager.cc:5495] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 1 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:32.493360 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:32.494480 18193 raft_consensus.cc:2454] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 0.
I20260807 03:38:32.495697 18576 raft_consensus.cc:482] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:32.496429 18249 leader_election.cc:302] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:32.496753 18655 raft_consensus.cc:482] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:32.497252 18655 raft_consensus.cc:504] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.497797 18576 raft_consensus.cc:504] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.499722 18655 leader_election.cc:288] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:32.499936 18576 leader_election.cc:288] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:32.500590 18639 raft_consensus.cc:2781] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:32.500964 18639 raft_consensus.cc:482] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:32.501281 18639 raft_consensus.cc:3037] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:32.501605 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:32.502029 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:32.502231 18190 raft_consensus.cc:2454] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 0.
I20260807 03:38:32.502897 18311 raft_consensus.cc:2379] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 1: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:38:32.503463 18027 leader_election.cc:302] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:32.504412 18576 raft_consensus.cc:2781] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:32.503759 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:32.504648 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:32.505072 18305 raft_consensus.cc:2454] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 0.
I20260807 03:38:32.506311 18402 raft_consensus.cc:482] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:32.506753 18402 raft_consensus.cc:504] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.507368 18369 tablet_bootstrap.cc:492] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:32.507774 18369 ts_tablet_manager.cc:1397] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.049s user 0.010s sys 0.000s
I20260807 03:38:32.508832 18402 leader_election.cc:288] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:32.509272 18365 tablet_bootstrap.cc:654] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:32.504776 18576 raft_consensus.cc:482] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:32.510953 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:32.511667 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:32.511827 18080 raft_consensus.cc:2454] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 0.
I20260807 03:38:32.512574 18193 raft_consensus.cc:2454] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 0.
I20260807 03:38:32.513329 18248 leader_election.cc:302] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:32.514297 18623 raft_consensus.cc:2781] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:32.514655 18623 raft_consensus.cc:482] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:32.514978 18623 raft_consensus.cc:3037] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:32.516824 18027 leader_election.cc:302] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:32.509266 18402 raft_consensus.cc:482] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:32.515837 18576 raft_consensus.cc:3037] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:32.524264 18638 raft_consensus.cc:482] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:32.525000 18366 tablet_bootstrap.cc:492] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:32.524770 18638 raft_consensus.cc:504] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.525571 18366 ts_tablet_manager.cc:1397] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.058s user 0.028s sys 0.016s
I20260807 03:38:32.518707 18402 raft_consensus.cc:504] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.527491 18638 leader_election.cc:288] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:32.524979 18369 raft_consensus.cc:348] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.529081 18369 raft_consensus.cc:374] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:32.528800 18402 leader_election.cc:288] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:32.529366 18369 raft_consensus.cc:729] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:32.530012 18369 consensus_queue.cc:260] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.530784 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:32.531634 18655 raft_consensus.cc:3037] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:32.532085 18369 ts_tablet_manager.cc:1428] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.024s user 0.005s sys 0.001s
W20260807 03:38:32.536059 18248 leader_election.cc:341] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:32.539479 18623 raft_consensus.cc:504] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.540843 18655 raft_consensus.cc:2726] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:32.533465 18366 raft_consensus.cc:348] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.541420 18623 leader_election.cc:288] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:32.541713 18366 raft_consensus.cc:374] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:32.542012 18366 raft_consensus.cc:729] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:32.543073 18369 tablet_bootstrap.cc:492] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:32.542613 18366 consensus_queue.cc:260] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.546236 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:32.546733 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:32.547122 18081 raft_consensus.cc:3037] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:32.547359 18080 raft_consensus.cc:2454] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 0.
I20260807 03:38:32.544288 18639 raft_consensus.cc:504] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.549998 18639 leader_election.cc:288] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:32.551620 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:32.552487 18190 raft_consensus.cc:2454] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 0.
I20260807 03:38:32.560532 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:32.564523 18249 leader_election.cc:302] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:32.565629 18248 leader_election.cc:302] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:32.566606 18623 raft_consensus.cc:2781] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:32.566934 18623 raft_consensus.cc:482] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:32.567317 18623 raft_consensus.cc:3037] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:32.572357 18366 ts_tablet_manager.cc:1428] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.047s user 0.004s sys 0.003s
I20260807 03:38:32.573315 18366 tablet_bootstrap.cc:492] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:32.572856 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:32.576485 18369 tablet_bootstrap.cc:654] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:32.576313 18623 raft_consensus.cc:504] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.577889 18639 raft_consensus.cc:2781] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:32.578289 18639 raft_consensus.cc:482] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:32.578617 18639 raft_consensus.cc:3037] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:32.579455 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:32.580072 18191 raft_consensus.cc:3037] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:32.583113 18193 raft_consensus.cc:2454] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 0.
I20260807 03:38:32.584731 18081 raft_consensus.cc:2454] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:32.589532 18638 consensus_queue.cc:1035] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:32.586443 18639 raft_consensus.cc:504] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.587627 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:32.592006 18402 consensus_queue.cc:1035] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:32.592334 18190 raft_consensus.cc:3037] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:32.598490 18576 raft_consensus.cc:504] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.607860 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:32.608476 18193 raft_consensus.cc:3037] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
W20260807 03:38:32.596038 18248 leader_election.cc:341] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:32.614185 18248 leader_election.cc:302] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:32.614964 18193 raft_consensus.cc:2454] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:38:32.615465 18638 raft_consensus.cc:2781] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:32.616133 18027 leader_election.cc:302] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:32.617043 18655 raft_consensus.cc:2781] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:32.594664 18191 raft_consensus.cc:2454] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:32.617539 18655 raft_consensus.cc:686] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:38:32.618459 18623 leader_election.cc:288] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:32.618397 18655 consensus_queue.cc:237] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.616138 18638 raft_consensus.cc:686] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:32.622514 18366 tablet_bootstrap.cc:654] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:32.623292 18638 consensus_queue.cc:237] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.629676 18075 raft_consensus.cc:3037] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:32.630141 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:32.632300 18639 leader_election.cc:288] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:32.636960 18365 tablet_bootstrap.cc:492] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:32.637401 18365 ts_tablet_manager.cc:1397] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.149s user 0.022s sys 0.049s
W20260807 03:38:32.641481 18248 leader_election.cc:341] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:32.640513 18190 raft_consensus.cc:2454] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:32.642390 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:32.642764 18690 raft_consensus.cc:482] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:32.642997 18305 raft_consensus.cc:3037] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:32.643190 18690 raft_consensus.cc:504] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.644490 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:32.648334 18082 raft_consensus.cc:3037] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
W20260807 03:38:32.638036 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 47715.7us
I20260807 03:38:32.650472 18365 raft_consensus.cc:348] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.651180 18365 raft_consensus.cc:374] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:32.651436 18365 raft_consensus.cc:729] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:32.651294 18249 leader_election.cc:302] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:32.652176 18365 consensus_queue.cc:260] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.645381 18576 leader_election.cc:288] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:32.661017 18365 ts_tablet_manager.cc:1428] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.023s user 0.007s sys 0.000s
I20260807 03:38:32.662011 18365 tablet_bootstrap.cc:492] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:32.676000 18619 raft_consensus.cc:482] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:32.676716 18619 raft_consensus.cc:504] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.654878 18639 raft_consensus.cc:2781] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:32.660974 18471 consensus_queue.cc:1035] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260807 03:38:32.655591 18305 raft_consensus.cc:2454] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:38:32.682809 18619 leader_election.cc:288] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:32.682705 18422 consensus_queue.cc:1035] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:32.686129 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:32.686776 18305 raft_consensus.cc:2454] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 0.
I20260807 03:38:32.686872 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:32.685192 18639 raft_consensus.cc:686] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:32.687512 18193 raft_consensus.cc:3037] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:32.688403 18141 leader_election.cc:302] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:32.688321 18639 consensus_queue.cc:237] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.693719 18422 raft_consensus.cc:2781] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:32.695132 18422 raft_consensus.cc:482] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:32.695444 18422 raft_consensus.cc:3037] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:32.697724 18082 raft_consensus.cc:2454] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:32.704435 18248 leader_election.cc:302] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:32.706153 18402 raft_consensus.cc:2781] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:32.700984 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:32.706606 18402 raft_consensus.cc:686] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:32.708252 18402 consensus_queue.cc:237] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.710683 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:32.711297 18190 raft_consensus.cc:3037] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:32.711313 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:32.711555 18193 raft_consensus.cc:2454] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:32.716190 18249 leader_election.cc:302] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
W20260807 03:38:32.716337 18248 leader_election.cc:341] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:32.717175 17920 catalog_manager.cc:5495] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 1 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
W20260807 03:38:32.713721 18140 leader_election.cc:341] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:32.724445 18623 raft_consensus.cc:2781] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:32.731027 18478 raft_consensus.cc:482] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:32.737474 18623 raft_consensus.cc:686] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:32.737637 18478 raft_consensus.cc:504] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.738334 18623 consensus_queue.cc:237] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.726471 18690 leader_election.cc:288] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:32.745118 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:32.746897 18365 tablet_bootstrap.cc:654] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:32.748533 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:32.752562 18027 leader_election.cc:302] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:32.753554 18576 raft_consensus.cc:2726] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:32.742148 18478 leader_election.cc:288] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:32.759044 18366 tablet_bootstrap.cc:492] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:32.759785 18366 ts_tablet_manager.cc:1397] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.187s user 0.034s sys 0.004s
I20260807 03:38:32.761263 18369 tablet_bootstrap.cc:492] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:32.761859 18369 ts_tablet_manager.cc:1397] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.219s user 0.011s sys 0.014s
I20260807 03:38:32.747256 18422 raft_consensus.cc:504] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.766041 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:32.767693 18190 raft_consensus.cc:2454] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:32.772761 18366 raft_consensus.cc:348] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:32.769665 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 53592.5us
I20260807 03:38:32.780728 18366 raft_consensus.cc:374] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:32.781785 18366 raft_consensus.cc:729] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:32.781081 18422 leader_election.cc:288] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:38:32.780462 18140 leader_election.cc:341] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:32.782503 18366 consensus_queue.cc:260] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.783859 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:32.788167 18311 raft_consensus.cc:3037] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:32.790019 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:32.790668 18190 raft_consensus.cc:2454] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 0.
W20260807 03:38:32.794799 18003 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 03:38:32.798497 18369 raft_consensus.cc:348] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.799222 18369 raft_consensus.cc:374] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:32.799522 18369 raft_consensus.cc:729] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:32.800197 18369 consensus_queue.cc:260] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.802330 18366 ts_tablet_manager.cc:1428] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.042s user 0.006s sys 0.000s
I20260807 03:38:32.803221 18366 tablet_bootstrap.cc:492] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:32.806388 18311 raft_consensus.cc:2454] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:32.807574 18249 leader_election.cc:302] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:32.814093 18141 leader_election.cc:302] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:32.815577 18619 raft_consensus.cc:2781] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:32.816048 18619 raft_consensus.cc:686] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:32.816447 18638 raft_consensus.cc:2781] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:32.816861 18638 raft_consensus.cc:482] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:32.817291 18638 raft_consensus.cc:3037] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:32.816972 18619 consensus_queue.cc:237] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.822196 18369 ts_tablet_manager.cc:1428] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.060s user 0.007s sys 0.000s
I20260807 03:38:32.823065 18369 tablet_bootstrap.cc:492] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:32.849639 17920 catalog_manager.cc:5495] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 0 to 1, leader changed from <none> to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 1 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:32.861325 18638 raft_consensus.cc:504] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.864918 18638 leader_election.cc:288] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:32.902650 18366 tablet_bootstrap.cc:654] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:32.890506 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:32.927186 17920 catalog_manager.cc:5495] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 1 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:38:32.933758 17920 catalog_manager.cc:5495] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 1 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:38:32.939369 17920 catalog_manager.cc:5495] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 1 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:38:32.935902 18191 raft_consensus.cc:3037] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:32.960563 18576 raft_consensus.cc:482] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:32.961899 18576 raft_consensus.cc:504] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:32.967265 18576 leader_election.cc:288] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:38:32.953342 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 77953us
I20260807 03:38:33.002888 18369 tablet_bootstrap.cc:654] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:33.005317 18191 raft_consensus.cc:2454] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:33.012439 18365 tablet_bootstrap.cc:492] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:33.012651 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:33.012897 18365 ts_tablet_manager.cc:1397] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.351s user 0.013s sys 0.015s
I20260807 03:38:33.015241 18365 raft_consensus.cc:348] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.014981 17915 catalog_manager.cc:5495] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 1 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:33.015939 18365 raft_consensus.cc:374] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:33.016362 18365 raft_consensus.cc:729] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:33.016571 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:33.017310 18193 raft_consensus.cc:2379] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 1: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:38:33.016981 18365 consensus_queue.cc:260] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.018510 18365 ts_tablet_manager.cc:1428] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260807 03:38:33.019308 18365 tablet_bootstrap.cc:492] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:33.029824 18632 consensus_queue.cc:1035] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:33.048138 18249 leader_election.cc:302] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:33.050360 18639 raft_consensus.cc:2781] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:33.051038 18639 raft_consensus.cc:686] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:33.052237 18639 consensus_queue.cc:237] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.035305 18027 leader_election.cc:302] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:33.061805 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
W20260807 03:38:33.064461 18248 leader_election.cc:341] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:33.068233 18576 raft_consensus.cc:3037] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
W20260807 03:38:33.069931 18140 consensus_peers.cc:487] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 -> Peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Couldn't send request to peer d9716e2bf06442f5b3b0e81087c53c4d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260807 03:38:33.075253 18576 raft_consensus.cc:2726] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:33.083822 18576 consensus_queue.cc:1035] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:33.091096 18366 tablet_bootstrap.cc:492] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:33.091526 18369 tablet_bootstrap.cc:492] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:33.091616 18366 ts_tablet_manager.cc:1397] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.289s user 0.022s sys 0.024s
I20260807 03:38:33.092298 18369 ts_tablet_manager.cc:1397] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.269s user 0.008s sys 0.006s
I20260807 03:38:33.098922 18366 raft_consensus.cc:348] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.099888 18366 raft_consensus.cc:374] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:33.100253 18366 raft_consensus.cc:729] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:33.099517 17915 catalog_manager.cc:5495] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 1 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:38:33.101475 18366 consensus_queue.cc:260] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.103592 18366 ts_tablet_manager.cc:1428] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20260807 03:38:33.105590 18365 tablet_bootstrap.cc:654] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:33.106468 18366 tablet_bootstrap.cc:492] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:33.107153 18369 raft_consensus.cc:348] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.107848 18369 raft_consensus.cc:374] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:33.108136 18369 raft_consensus.cc:729] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:33.108754 18369 consensus_queue.cc:260] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.110800 18369 ts_tablet_manager.cc:1428] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.018s user 0.003s sys 0.000s
I20260807 03:38:33.111920 18369 tablet_bootstrap.cc:492] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:33.127753 18366 tablet_bootstrap.cc:654] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:33.131114 18576 consensus_queue.cc:1035] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 03:38:33.134339 18369 tablet_bootstrap.cc:654] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:33.138253 18638 consensus_queue.cc:1035] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:33.148317 18365 tablet_bootstrap.cc:492] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:33.148838 18365 ts_tablet_manager.cc:1397] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.130s user 0.014s sys 0.013s
I20260807 03:38:33.150496 18632 raft_consensus.cc:482] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:33.151005 18632 raft_consensus.cc:504] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.153052 18632 leader_election.cc:288] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:33.154561 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:33.154945 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:38:33.156438 18141 leader_election.cc:341] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021): Illegal state: must be running to vote when last-logged opid is not known
W20260807 03:38:33.157222 18140 leader_election.cc:341] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:33.157723 18140 leader_election.cc:302] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
W20260807 03:38:33.175621 18248 consensus_peers.cc:487] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 -> Peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Couldn't send request to peer d9716e2bf06442f5b3b0e81087c53c4d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260807 03:38:33.190580 18365 raft_consensus.cc:348] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.191442 18365 raft_consensus.cc:374] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:33.191711 18365 raft_consensus.cc:729] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:33.190582 18632 raft_consensus.cc:2726] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:33.192323 18365 consensus_queue.cc:260] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.194166 18365 ts_tablet_manager.cc:1428] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.045s user 0.001s sys 0.003s
I20260807 03:38:33.195029 18365 tablet_bootstrap.cc:492] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:33.217408 18366 tablet_bootstrap.cc:492] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:33.217798 18366 ts_tablet_manager.cc:1397] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.111s user 0.027s sys 0.011s
I20260807 03:38:33.220534 18365 tablet_bootstrap.cc:654] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:33.221014 18366 raft_consensus.cc:348] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.221769 18366 raft_consensus.cc:374] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:33.222043 18366 raft_consensus.cc:729] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:33.222687 18366 consensus_queue.cc:260] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.223062 18402 consensus_queue.cc:1035] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:33.223770 18638 consensus_queue.cc:1035] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:33.225450 18402 consensus_queue.cc:1035] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:33.229803 18081 raft_consensus.cc:3037] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:33.244652 18632 consensus_queue.cc:1035] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260807 03:38:33.249284 18366 ts_tablet_manager.cc:1428] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.031s user 0.005s sys 0.001s
I20260807 03:38:33.250172 18366 tablet_bootstrap.cc:492] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:33.255503 18369 tablet_bootstrap.cc:492] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:33.256028 18369 ts_tablet_manager.cc:1397] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.144s user 0.010s sys 0.005s
I20260807 03:38:33.258625 18366 tablet_bootstrap.cc:654] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:33.265066 18639 consensus_queue.cc:1035] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:33.266155 18369 raft_consensus.cc:348] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.266974 18369 raft_consensus.cc:374] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:33.267254 18369 raft_consensus.cc:729] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:33.268438 18369 consensus_queue.cc:260] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.270588 18369 ts_tablet_manager.cc:1428] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.014s user 0.004s sys 0.000s
I20260807 03:38:33.271481 18369 tablet_bootstrap.cc:492] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:33.297636 18369 tablet_bootstrap.cc:654] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:33.300414 18632 consensus_queue.cc:1035] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 03:38:33.307452 18366 tablet_bootstrap.cc:492] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:33.307928 18366 ts_tablet_manager.cc:1397] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.058s user 0.012s sys 0.035s
I20260807 03:38:33.310554 18366 raft_consensus.cc:348] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.311463 18366 raft_consensus.cc:374] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:33.311797 18366 raft_consensus.cc:729] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:33.312737 18366 consensus_queue.cc:260] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.315491 18366 ts_tablet_manager.cc:1428] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260807 03:38:33.316581 18366 tablet_bootstrap.cc:492] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:33.323810 18366 tablet_bootstrap.cc:654] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:33.332706 18402 consensus_queue.cc:1035] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 03:38:33.337170 18366 tablet_bootstrap.cc:492] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:33.337587 18366 ts_tablet_manager.cc:1397] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.021s user 0.012s sys 0.004s
I20260807 03:38:33.339519 18638 consensus_queue.cc:1035] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:33.339905 18366 raft_consensus.cc:348] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.340963 18639 raft_consensus.cc:482] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:33.343763 18623 raft_consensus.cc:482] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:33.344216 18366 raft_consensus.cc:374] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:33.344697 18366 raft_consensus.cc:729] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:33.344411 18623 raft_consensus.cc:504] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.345533 18366 consensus_queue.cc:260] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.346580 18623 leader_election.cc:288] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:33.347803 18366 ts_tablet_manager.cc:1428] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260807 03:38:33.347337 18639 raft_consensus.cc:504] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.348483 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:33.351015 18366 tablet_bootstrap.cc:492] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:33.351081 18369 tablet_bootstrap.cc:492] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:33.351589 18369 ts_tablet_manager.cc:1397] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.080s user 0.013s sys 0.014s
I20260807 03:38:33.352368 18639 leader_election.cc:288] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:33.354370 18369 raft_consensus.cc:348] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.355038 18369 raft_consensus.cc:374] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:33.355330 18369 raft_consensus.cc:729] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:33.356117 18369 consensus_queue.cc:260] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.356802 18366 tablet_bootstrap.cc:654] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:33.357589 18369 ts_tablet_manager.cc:1428] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260807 03:38:33.357563 18249 leader_election.cc:341] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:33.358697 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:33.359345 18188 raft_consensus.cc:2454] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 0.
I20260807 03:38:33.360538 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:33.361243 18073 raft_consensus.cc:2454] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 0.
I20260807 03:38:33.361995 18249 leader_election.cc:302] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:33.364665 18369 tablet_bootstrap.cc:492] T af8877c3fbba4ab18765832f1390921f P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:33.364646 18639 raft_consensus.cc:2781] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:33.365444 18639 raft_consensus.cc:482] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:33.365965 18639 raft_consensus.cc:3037] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:33.370805 18369 tablet_bootstrap.cc:654] T af8877c3fbba4ab18765832f1390921f P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:33.375840 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:33.401778 18639 raft_consensus.cc:504] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.403748 18639 leader_election.cc:288] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:33.420508 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:33.419016 18248 leader_election.cc:302] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:38:33.424281 18193 raft_consensus.cc:3037] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:33.424553 18402 raft_consensus.cc:2781] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:33.425318 18402 raft_consensus.cc:482] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:33.425691 18402 raft_consensus.cc:3037] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:33.431644 18655 raft_consensus.cc:482] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 03:38:33.431699 18248 leader_election.cc:341] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:33.441480 18369 tablet_bootstrap.cc:492] T af8877c3fbba4ab18765832f1390921f P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:33.441897 18369 ts_tablet_manager.cc:1397] T af8877c3fbba4ab18765832f1390921f P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.077s user 0.009s sys 0.017s
I20260807 03:38:33.432368 18655 raft_consensus.cc:504] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.447700 18655 leader_election.cc:288] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:33.449038 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:33.451011 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:33.451450 18193 raft_consensus.cc:2454] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:33.451838 18190 raft_consensus.cc:2379] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 1: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:38:33.452965 18027 leader_election.cc:302] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:33.453784 18655 raft_consensus.cc:3037] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:33.454222 18249 leader_election.cc:302] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:33.456944 18365 tablet_bootstrap.cc:492] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:33.457353 18365 ts_tablet_manager.cc:1397] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.263s user 0.044s sys 0.052s
I20260807 03:38:33.461874 18638 raft_consensus.cc:2781] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:33.462460 18638 raft_consensus.cc:686] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:33.469177 18402 raft_consensus.cc:504] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.463454 18638 consensus_queue.cc:237] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.492702 18365 raft_consensus.cc:348] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.493852 18365 raft_consensus.cc:374] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:33.494164 18365 raft_consensus.cc:729] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:33.496585 18365 consensus_queue.cc:260] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.504031 18402 leader_election.cc:288] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:33.506417 18365 ts_tablet_manager.cc:1428] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.049s user 0.009s sys 0.017s
I20260807 03:38:33.507144 18365 tablet_bootstrap.cc:492] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:33.505822 18369 raft_consensus.cc:348] T af8877c3fbba4ab18765832f1390921f P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.508023 18369 raft_consensus.cc:374] T af8877c3fbba4ab18765832f1390921f P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:33.508378 18369 raft_consensus.cc:729] T af8877c3fbba4ab18765832f1390921f P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:33.509383 18369 consensus_queue.cc:260] T af8877c3fbba4ab18765832f1390921f P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:33.496336 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 55153.6us
I20260807 03:38:33.504594 18402 raft_consensus.cc:482] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:33.516559 18366 tablet_bootstrap.cc:492] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:33.517351 18366 ts_tablet_manager.cc:1397] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.167s user 0.026s sys 0.025s
I20260807 03:38:33.517789 18402 raft_consensus.cc:504] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.521384 18402 leader_election.cc:288] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:33.521272 18366 raft_consensus.cc:348] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.522035 18366 raft_consensus.cc:374] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:33.522307 18366 raft_consensus.cc:729] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
W20260807 03:38:33.514683 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 60535.8us
I20260807 03:38:33.522866 18366 consensus_queue.cc:260] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.526343 18369 ts_tablet_manager.cc:1428] T af8877c3fbba4ab18765832f1390921f P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.084s user 0.007s sys 0.002s
I20260807 03:38:33.527119 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:33.527339 18369 tablet_bootstrap.cc:492] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:33.527671 18191 raft_consensus.cc:3037] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:33.532145 18366 ts_tablet_manager.cc:1428] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20260807 03:38:33.533157 18366 tablet_bootstrap.cc:492] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:33.539335 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:33.540936 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:33.542261 18080 raft_consensus.cc:3037] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:33.568630 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:33.571144 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
W20260807 03:38:33.571768 18249 leader_election.cc:341] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:33.571995 18081 raft_consensus.cc:3037] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:33.574762 18632 raft_consensus.cc:482] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:33.575310 18632 raft_consensus.cc:504] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.576514 18369 tablet_bootstrap.cc:654] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:33.577457 18632 leader_election.cc:288] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:33.578591 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:33.579257 18312 raft_consensus.cc:2454] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 0.
I20260807 03:38:33.582617 18365 tablet_bootstrap.cc:654] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
W20260807 03:38:33.587495 18248 leader_election.cc:341] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:33.588300 18366 tablet_bootstrap.cc:654] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
W20260807 03:38:33.578613 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 53944.3us
I20260807 03:38:33.597361 18655 raft_consensus.cc:2726] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:33.601467 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:33.594605 18141 leader_election.cc:302] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:33.583472 18191 raft_consensus.cc:2454] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:33.608783 18619 raft_consensus.cc:2781] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:33.610280 18619 raft_consensus.cc:482] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 03:38:33.600970 18248 leader_election.cc:341] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:33.611934 18619 raft_consensus.cc:3037] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
W20260807 03:38:33.616501 18140 leader_election.cc:341] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:33.638654 18249 leader_election.cc:302] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:33.639611 18757 raft_consensus.cc:2781] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:33.640066 18757 raft_consensus.cc:686] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:33.640846 18757 consensus_queue.cc:237] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.643365 18081 raft_consensus.cc:2454] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:33.646018 18619 raft_consensus.cc:504] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.620921 18248 leader_election.cc:302] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:33.649114 18756 raft_consensus.cc:2726] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260807 03:38:33.649547 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 53832.1us
I20260807 03:38:33.651423 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:33.652060 18369 tablet_bootstrap.cc:492] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:33.652619 18369 ts_tablet_manager.cc:1397] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.125s user 0.015s sys 0.021s
W20260807 03:38:33.652688 18140 leader_election.cc:341] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:33.656018 18756 raft_consensus.cc:482] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:33.656172 18369 raft_consensus.cc:348] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.656893 18369 raft_consensus.cc:374] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:33.656523 18756 raft_consensus.cc:504] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.657163 18369 raft_consensus.cc:729] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:33.657812 18369 consensus_queue.cc:260] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.658666 18756 leader_election.cc:288] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:33.659518 18369 ts_tablet_manager.cc:1428] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20260807 03:38:33.660640 18756 consensus_queue.cc:1035] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:33.673106 18369 tablet_bootstrap.cc:492] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:33.673177 18619 leader_election.cc:288] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:33.674275 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:33.674865 18312 raft_consensus.cc:3037] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:33.682771 18365 tablet_bootstrap.cc:492] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:33.683225 18365 ts_tablet_manager.cc:1397] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.176s user 0.010s sys 0.006s
I20260807 03:38:33.685111 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:33.688594 17920 catalog_manager.cc:5495] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 1 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:38:33.690552 17920 catalog_manager.cc:5495] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 1 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:38:33.692795 18312 raft_consensus.cc:2454] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:33.693914 18141 leader_election.cc:302] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:33.695402 18632 raft_consensus.cc:2781] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:33.695964 18632 raft_consensus.cc:686] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:33.696962 18632 consensus_queue.cc:237] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.730500 18757 consensus_queue.cc:1035] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260807 03:38:33.757706 18365 raft_consensus.cc:348] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.758646 18365 raft_consensus.cc:374] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:33.758924 18365 raft_consensus.cc:729] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
W20260807 03:38:33.759053 18248 leader_election.cc:341] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:33.759516 18365 consensus_queue.cc:260] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.767401 18369 tablet_bootstrap.cc:654] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:33.767835 18366 tablet_bootstrap.cc:492] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:33.768476 18366 ts_tablet_manager.cc:1397] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.235s user 0.021s sys 0.008s
I20260807 03:38:33.771026 18366 raft_consensus.cc:348] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.771705 18366 raft_consensus.cc:374] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:33.771962 18366 raft_consensus.cc:729] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:33.775354 18366 consensus_queue.cc:260] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.770704 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:33.778286 18193 raft_consensus.cc:2454] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 0.
I20260807 03:38:33.779456 18249 leader_election.cc:302] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:33.780462 18366 ts_tablet_manager.cc:1428] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260807 03:38:33.780915 18753 raft_consensus.cc:2781] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:33.783308 18753 raft_consensus.cc:482] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:33.783667 18753 raft_consensus.cc:3037] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:33.784600 18771 raft_consensus.cc:482] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:33.786038 17912 catalog_manager.cc:5495] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 1 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:33.787029 18366 tablet_bootstrap.cc:492] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:33.785133 18771 raft_consensus.cc:504] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.761855 18365 ts_tablet_manager.cc:1428] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.078s user 0.007s sys 0.000s
I20260807 03:38:33.795445 18365 tablet_bootstrap.cc:492] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:33.803588 18771 leader_election.cc:288] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:33.813867 18753 raft_consensus.cc:504] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.817780 18753 leader_election.cc:288] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:33.819919 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:33.823449 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:33.823448 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:33.824931 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:33.826563 18312 raft_consensus.cc:2454] T af8877c3fbba4ab18765832f1390921f P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 0.
I20260807 03:38:33.827220 18080 raft_consensus.cc:3037] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:33.827630 18369 tablet_bootstrap.cc:492] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:33.828058 18369 ts_tablet_manager.cc:1397] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.155s user 0.014s sys 0.011s
I20260807 03:38:33.830107 18366 tablet_bootstrap.cc:654] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:33.829902 18369 raft_consensus.cc:348] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.830446 18369 raft_consensus.cc:374] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:33.830674 18369 raft_consensus.cc:729] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:33.831043 18193 raft_consensus.cc:3037] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:33.831718 18141 leader_election.cc:302] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:33.832587 18632 raft_consensus.cc:2781] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:33.832931 18632 raft_consensus.cc:482] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:33.833423 18632 raft_consensus.cc:3037] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:33.831411 18369 consensus_queue.cc:260] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:33.836735 18140 leader_election.cc:341] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:33.837359 18080 raft_consensus.cc:2454] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:33.841107 18369 ts_tablet_manager.cc:1428] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.013s user 0.005s sys 0.001s
I20260807 03:38:33.842229 18369 tablet_bootstrap.cc:492] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:33.846019 18632 raft_consensus.cc:504] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.847055 18365 tablet_bootstrap.cc:654] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:33.849433 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:33.851744 18312 raft_consensus.cc:3037] T af8877c3fbba4ab18765832f1390921f P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:33.847949 18632 leader_election.cc:288] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:33.854666 18248 leader_election.cc:302] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:33.856392 18753 raft_consensus.cc:2781] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:33.856846 18753 raft_consensus.cc:686] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:33.857606 18753 consensus_queue.cc:237] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.869170 18312 raft_consensus.cc:2454] T af8877c3fbba4ab18765832f1390921f P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:33.871155 18141 leader_election.cc:302] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:33.876302 18775 raft_consensus.cc:2781] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:33.876798 18775 raft_consensus.cc:686] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:33.876340 18366 tablet_bootstrap.cc:492] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:33.877555 18775 consensus_queue.cc:237] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.879344 18366 ts_tablet_manager.cc:1397] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.092s user 0.012s sys 0.014s
I20260807 03:38:33.883472 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:33.884200 18193 raft_consensus.cc:2454] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:33.888726 18369 tablet_bootstrap.cc:654] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
W20260807 03:38:33.891819 18140 leader_election.cc:341] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:33.900694 17912 catalog_manager.cc:5495] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 1 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:38:33.902232 18366 raft_consensus.cc:348] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.905447 18366 raft_consensus.cc:374] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:33.905673 18366 raft_consensus.cc:729] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:33.911054 18366 consensus_queue.cc:260] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.914831 18760 raft_consensus.cc:482] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:33.915305 18760 raft_consensus.cc:504] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.916836 18366 ts_tablet_manager.cc:1428] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.035s user 0.007s sys 0.002s
I20260807 03:38:33.917793 18760 leader_election.cc:288] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:33.917968 18366 tablet_bootstrap.cc:492] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:33.922168 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:33.922353 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:33.924185 18312 raft_consensus.cc:2454] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 0.
W20260807 03:38:33.924871 18027 leader_election.cc:341] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:33.925576 18027 leader_election.cc:302] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:38:33.926345 18760 raft_consensus.cc:2781] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:33.926816 18760 raft_consensus.cc:482] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:33.927271 18760 raft_consensus.cc:3037] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:33.939177 18365 tablet_bootstrap.cc:492] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:33.939616 18365 ts_tablet_manager.cc:1397] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.144s user 0.014s sys 0.002s
I20260807 03:38:33.941154 17914 catalog_manager.cc:5495] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 1 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:33.945256 18366 tablet_bootstrap.cc:654] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:33.946233 18365 raft_consensus.cc:348] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.946709 18365 raft_consensus.cc:374] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:33.946866 18365 raft_consensus.cc:729] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
W20260807 03:38:33.947703 18248 consensus_peers.cc:487] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 -> Peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Couldn't send request to peer d9716e2bf06442f5b3b0e81087c53c4d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260807 03:38:33.950040 18369 tablet_bootstrap.cc:492] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:33.949186 18365 consensus_queue.cc:260] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.950456 18369 ts_tablet_manager.cc:1397] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.108s user 0.012s sys 0.004s
I20260807 03:38:33.956661 18369 raft_consensus.cc:348] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.957145 18369 raft_consensus.cc:374] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:33.957427 18369 raft_consensus.cc:729] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:33.957875 18369 consensus_queue.cc:260] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.961634 18365 ts_tablet_manager.cc:1428] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.022s user 0.005s sys 0.001s
I20260807 03:38:33.964383 18369 ts_tablet_manager.cc:1428] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.014s user 0.006s sys 0.001s
I20260807 03:38:33.965180 18760 raft_consensus.cc:504] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.966291 18365 tablet_bootstrap.cc:492] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:33.967144 18760 leader_election.cc:288] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:33.968691 18369 tablet_bootstrap.cc:492] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:33.968783 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:33.969182 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:33.969786 18312 raft_consensus.cc:3037] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
W20260807 03:38:33.970147 18027 leader_election.cc:341] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:33.975823 18369 tablet_bootstrap.cc:654] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:33.977542 18756 consensus_queue.cc:1035] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:33.985020 18365 tablet_bootstrap.cc:654] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
W20260807 03:38:33.985579 18003 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 03:38:33.988786 18366 tablet_bootstrap.cc:492] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:33.992002 18366 ts_tablet_manager.cc:1397] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.074s user 0.012s sys 0.003s
I20260807 03:38:33.994931 18366 raft_consensus.cc:348] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:33.995643 18366 raft_consensus.cc:374] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:33.995920 18366 raft_consensus.cc:729] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:33.996541 18366 consensus_queue.cc:260] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.001518 18366 ts_tablet_manager.cc:1428] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260807 03:38:34.009092 18366 tablet_bootstrap.cc:492] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:34.017540 18366 tablet_bootstrap.cc:654] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:34.023846 18365 tablet_bootstrap.cc:492] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:34.024482 18365 ts_tablet_manager.cc:1397] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.058s user 0.013s sys 0.000s
I20260807 03:38:34.034497 18478 raft_consensus.cc:482] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:34.034981 18478 raft_consensus.cc:504] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.035454 18365 raft_consensus.cc:348] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.036425 18365 raft_consensus.cc:374] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:34.036743 18365 raft_consensus.cc:729] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:34.037752 18478 leader_election.cc:288] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:34.038928 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:34.039575 18191 raft_consensus.cc:2454] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 0.
I20260807 03:38:34.037487 18365 consensus_queue.cc:260] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.040889 18027 leader_election.cc:302] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:34.042627 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:34.042964 18365 ts_tablet_manager.cc:1428] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.018s user 0.008s sys 0.000s
W20260807 03:38:34.044037 18027 leader_election.cc:341] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:34.044947 18478 raft_consensus.cc:2781] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:34.045323 18478 raft_consensus.cc:482] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:34.045719 18478 raft_consensus.cc:3037] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:34.054124 18365 tablet_bootstrap.cc:492] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:34.064227 18366 tablet_bootstrap.cc:492] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:34.070127 18366 ts_tablet_manager.cc:1397] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.061s user 0.012s sys 0.007s
I20260807 03:38:34.073690 18312 raft_consensus.cc:2454] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:38:34.073952 18366 raft_consensus.cc:348] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.074930 18366 raft_consensus.cc:374] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:34.075253 18366 raft_consensus.cc:729] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:34.078285 18365 tablet_bootstrap.cc:654] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:34.080348 18027 leader_election.cc:302] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:38:34.081218 18760 raft_consensus.cc:2781] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:34.081696 18760 raft_consensus.cc:686] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:38:34.082594 18760 consensus_queue.cc:237] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.083727 18366 consensus_queue.cc:260] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.088999 18366 ts_tablet_manager.cc:1428] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.018s user 0.009s sys 0.000s
I20260807 03:38:34.090231 18366 tablet_bootstrap.cc:492] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:34.093858 18365 tablet_bootstrap.cc:492] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:34.094491 18365 ts_tablet_manager.cc:1397] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.041s user 0.014s sys 0.000s
I20260807 03:38:34.095808 18753 raft_consensus.cc:482] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:34.096452 18753 raft_consensus.cc:504] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.101893 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:38:34.103936 18249 leader_election.cc:341] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:34.105772 18753 leader_election.cc:288] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:34.106802 18632 raft_consensus.cc:482] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:34.107546 18632 raft_consensus.cc:504] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.113930 18632 leader_election.cc:288] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:34.116130 18365 raft_consensus.cc:348] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.117064 18365 raft_consensus.cc:374] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:34.117381 18365 raft_consensus.cc:729] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:34.121703 18366 tablet_bootstrap.cc:654] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:34.121764 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:34.122757 18312 raft_consensus.cc:2379] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 1: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:38:34.124449 18365 consensus_queue.cc:260] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.127739 18365 ts_tablet_manager.cc:1428] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.033s user 0.007s sys 0.000s
I20260807 03:38:34.130239 18365 tablet_bootstrap.cc:492] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:34.132493 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:38:34.133683 18248 leader_election.cc:341] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:34.134087 18248 leader_election.cc:302] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:34.135329 18753 raft_consensus.cc:2726] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:34.136827 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:34.139400 18140 leader_election.cc:302] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:34.140422 18632 raft_consensus.cc:3037] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:34.143854 18756 raft_consensus.cc:482] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:34.144382 18756 raft_consensus.cc:504] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.146471 18756 leader_election.cc:288] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:34.150528 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:38:34.152930 18249 leader_election.cc:341] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:34.165285 18365 tablet_bootstrap.cc:654] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:34.168185 18775 consensus_queue.cc:1035] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:34.171769 18366 tablet_bootstrap.cc:492] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:34.173022 18366 ts_tablet_manager.cc:1397] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.083s user 0.016s sys 0.000s
I20260807 03:38:34.176371 18366 raft_consensus.cc:348] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.177264 18366 raft_consensus.cc:374] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:34.177593 18366 raft_consensus.cc:729] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:34.178352 18366 consensus_queue.cc:260] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.185966 18365 tablet_bootstrap.cc:492] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:34.186532 18365 ts_tablet_manager.cc:1397] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.057s user 0.006s sys 0.006s
I20260807 03:38:34.189782 18366 ts_tablet_manager.cc:1428] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.016s user 0.008s sys 0.000s
I20260807 03:38:34.191588 18366 tablet_bootstrap.cc:492] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:34.196458 18080 raft_consensus.cc:3037] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:34.192341 18365 raft_consensus.cc:348] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.203758 18365 raft_consensus.cc:374] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:34.204018 18365 raft_consensus.cc:729] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:34.206239 18365 consensus_queue.cc:260] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.207319 18760 raft_consensus.cc:482] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:34.207808 18760 raft_consensus.cc:504] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.209724 18760 leader_election.cc:288] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:34.211167 18366 tablet_bootstrap.cc:654] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:34.131927 18478 raft_consensus.cc:504] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.215109 18632 raft_consensus.cc:2726] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:34.218097 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:34.218436 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:34.219913 18027 leader_election.cc:302] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:34.220731 18760 raft_consensus.cc:2726] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:34.221071 18365 ts_tablet_manager.cc:1428] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.034s user 0.006s sys 0.000s
I20260807 03:38:34.221954 18365 tablet_bootstrap.cc:492] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:34.230171 17915 catalog_manager.cc:5495] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 0 to 1, leader changed from <none> to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 1 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:34.233832 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:34.235270 18478 leader_election.cc:288] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:38:34.238430 18248 leader_election.cc:341] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:34.238863 18248 leader_election.cc:302] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:34.239634 18757 raft_consensus.cc:2726] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:34.241386 18075 raft_consensus.cc:3037] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:34.247648 18365 tablet_bootstrap.cc:654] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:34.248764 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:34.252722 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:34.253324 18193 raft_consensus.cc:3037] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:34.267119 18369 tablet_bootstrap.cc:492] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:34.267565 18369 ts_tablet_manager.cc:1397] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.299s user 0.084s sys 0.125s
I20260807 03:38:34.273175 18369 raft_consensus.cc:348] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.273234 18471 consensus_queue.cc:1035] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 03:38:34.273893 18369 raft_consensus.cc:374] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:34.274184 18369 raft_consensus.cc:729] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:34.274782 18369 consensus_queue.cc:260] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.282632 18369 ts_tablet_manager.cc:1428] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.015s user 0.008s sys 0.000s
I20260807 03:38:34.283504 18369 tablet_bootstrap.cc:492] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:34.283543 18756 consensus_queue.cc:1035] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
W20260807 03:38:34.289877 18027 leader_election.cc:341] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:34.298329 18623 consensus_queue.cc:1035] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20260807 03:38:34.350625 18366 tablet_bootstrap.cc:492] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:34.351140 18366 ts_tablet_manager.cc:1397] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.161s user 0.015s sys 0.015s
I20260807 03:38:34.361763 18365 tablet_bootstrap.cc:492] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:34.362246 18365 ts_tablet_manager.cc:1397] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.140s user 0.012s sys 0.000s
I20260807 03:38:34.349193 18193 raft_consensus.cc:2454] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:38:34.360296 18366 raft_consensus.cc:348] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.365093 18366 raft_consensus.cc:374] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:34.365386 18366 raft_consensus.cc:729] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:34.365933 18366 consensus_queue.cc:260] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.366118 18365 raft_consensus.cc:348] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.366734 18365 raft_consensus.cc:374] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:34.366999 18365 raft_consensus.cc:729] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:34.366451 18369 tablet_bootstrap.cc:654] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:34.368055 18366 ts_tablet_manager.cc:1428] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.017s user 0.004s sys 0.000s
I20260807 03:38:34.368322 18365 consensus_queue.cc:260] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.368934 18366 tablet_bootstrap.cc:492] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:34.376870 18027 leader_election.cc:302] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:34.378998 18365 ts_tablet_manager.cc:1428] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20260807 03:38:34.379783 18365 tablet_bootstrap.cc:492] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:34.384479 18775 raft_consensus.cc:482] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:34.380577 18760 raft_consensus.cc:2781] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:34.384886 18775 raft_consensus.cc:504] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.386121 18760 raft_consensus.cc:686] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:38:34.386969 18760 consensus_queue.cc:237] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.387562 18775 leader_election.cc:288] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:34.383438 18757 consensus_queue.cc:1035] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20260807 03:38:34.415055 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:34.428650 18366 tablet_bootstrap.cc:654] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:34.430538 18369 tablet_bootstrap.cc:492] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:34.430967 18369 ts_tablet_manager.cc:1397] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.148s user 0.011s sys 0.004s
W20260807 03:38:34.440600 18140 leader_election.cc:341] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:34.440778 17915 catalog_manager.cc:5495] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 0 to 1, leader changed from <none> to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 1 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:34.440547 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:34.443307 18632 consensus_queue.cc:1035] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 03:38:34.443645 18141 leader_election.cc:341] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:34.444087 18141 leader_election.cc:302] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:34.444324 18365 tablet_bootstrap.cc:654] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:34.446241 18369 raft_consensus.cc:348] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.450816 18369 raft_consensus.cc:374] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:34.451063 18369 raft_consensus.cc:729] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:34.451913 18756 consensus_queue.cc:1035] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 03:38:34.454324 18369 consensus_queue.cc:260] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.450301 18619 raft_consensus.cc:2726] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260807 03:38:34.461855 18140 consensus_peers.cc:487] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 -> Peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Couldn't send request to peer d9716e2bf06442f5b3b0e81087c53c4d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260807 03:38:34.469445 18079 raft_consensus.cc:3037] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:34.470920 18369 ts_tablet_manager.cc:1428] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.040s user 0.004s sys 0.003s
I20260807 03:38:34.471871 18369 tablet_bootstrap.cc:492] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:34.475001 18619 raft_consensus.cc:482] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:34.475484 18619 raft_consensus.cc:504] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.477626 18619 leader_election.cc:288] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:34.480003 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:38:34.488724 18140 leader_election.cc:341] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:34.496068 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:34.496531 18760 consensus_queue.cc:1035] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20260807 03:38:34.512657 18366 tablet_bootstrap.cc:492] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:34.513252 18366 ts_tablet_manager.cc:1397] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.144s user 0.014s sys 0.016s
W20260807 03:38:34.506872 18141 leader_election.cc:341] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:34.516397 18141 leader_election.cc:302] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:34.517442 18619 raft_consensus.cc:2726] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:34.518276 18366 raft_consensus.cc:348] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.519205 18366 raft_consensus.cc:374] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:34.519490 18366 raft_consensus.cc:729] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:34.525103 18366 consensus_queue.cc:260] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.540171 18366 ts_tablet_manager.cc:1428] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.027s user 0.008s sys 0.000s
I20260807 03:38:34.541388 18366 tablet_bootstrap.cc:492] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:34.547432 18369 tablet_bootstrap.cc:654] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:34.552884 18365 tablet_bootstrap.cc:492] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:34.553421 18365 ts_tablet_manager.cc:1397] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.174s user 0.011s sys 0.026s
I20260807 03:38:34.566404 18366 tablet_bootstrap.cc:654] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:34.574468 18365 raft_consensus.cc:348] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.575079 18365 raft_consensus.cc:374] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:34.575294 18365 raft_consensus.cc:729] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:34.575783 18365 consensus_queue.cc:260] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.583448 18655 consensus_queue.cc:1035] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:34.589496 18365 ts_tablet_manager.cc:1428] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.036s user 0.006s sys 0.000s
I20260807 03:38:34.590600 18365 tablet_bootstrap.cc:492] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:34.609383 18623 raft_consensus.cc:482] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:34.609889 18623 raft_consensus.cc:504] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.614089 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:34.612004 18623 leader_election.cc:288] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:34.615870 18655 raft_consensus.cc:482] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:34.617169 18655 raft_consensus.cc:504] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.619019 18655 leader_election.cc:288] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:34.619529 18365 tablet_bootstrap.cc:654] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:34.622372 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:34.623176 18190 raft_consensus.cc:2379] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 1: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:38:34.622751 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:34.629176 18249 leader_election.cc:302] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:34.629899 18369 tablet_bootstrap.cc:492] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:34.630285 18753 raft_consensus.cc:3037] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:34.630425 18369 ts_tablet_manager.cc:1397] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.159s user 0.011s sys 0.006s
I20260807 03:38:34.634232 18369 raft_consensus.cc:348] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:34.635251 18027 leader_election.cc:341] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:34.635511 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:34.645763 18305 raft_consensus.cc:2454] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 0.
I20260807 03:38:34.642758 18366 tablet_bootstrap.cc:492] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:34.646977 18366 ts_tablet_manager.cc:1397] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.106s user 0.008s sys 0.005s
I20260807 03:38:34.647367 18027 leader_election.cc:302] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:38:34.648504 18478 raft_consensus.cc:482] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:34.648191 18369 raft_consensus.cc:374] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:34.649298 18369 raft_consensus.cc:729] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:34.650316 18655 raft_consensus.cc:2781] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:34.649025 18478 raft_consensus.cc:504] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.650950 18655 raft_consensus.cc:482] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:34.651438 18655 raft_consensus.cc:3037] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:34.652480 18478 leader_election.cc:288] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:34.652489 18619 raft_consensus.cc:482] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:34.653316 18619 raft_consensus.cc:504] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.653837 18753 raft_consensus.cc:2726] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:34.655195 18619 leader_election.cc:288] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:34.656296 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:34.656723 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:38:34.657594 18141 leader_election.cc:341] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021): Illegal state: must be running to vote when last-logged opid is not known
W20260807 03:38:34.658047 18140 leader_election.cc:341] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:34.658417 18140 leader_election.cc:302] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:34.659422 18366 raft_consensus.cc:348] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.660118 18366 raft_consensus.cc:374] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:34.660182 18632 raft_consensus.cc:2726] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:34.660785 18366 raft_consensus.cc:729] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:34.660449 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:34.663411 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:34.664316 18193 raft_consensus.cc:2454] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 0.
W20260807 03:38:34.663393 18027 leader_election.cc:341] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:34.661464 18366 consensus_queue.cc:260] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.667743 18366 ts_tablet_manager.cc:1428] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.019s user 0.005s sys 0.000s
I20260807 03:38:34.668651 18027 leader_election.cc:302] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:34.669752 18478 raft_consensus.cc:2781] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:34.670126 18478 raft_consensus.cc:482] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:34.670446 18478 raft_consensus.cc:3037] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:34.673202 18655 raft_consensus.cc:504] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.674981 18366 tablet_bootstrap.cc:492] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:34.674472 18369 consensus_queue.cc:260] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.680948 18369 ts_tablet_manager.cc:1428] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.050s user 0.005s sys 0.002s
I20260807 03:38:34.684638 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:34.688457 18478 raft_consensus.cc:504] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.682659 18369 tablet_bootstrap.cc:492] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:34.699003 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:34.699541 18305 raft_consensus.cc:3037] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
W20260807 03:38:34.700793 18027 leader_election.cc:341] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:34.704124 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:34.704954 18191 raft_consensus.cc:3037] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:34.680509 18655 leader_election.cc:288] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:34.715124 18305 raft_consensus.cc:2454] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:38:34.717612 18027 leader_election.cc:302] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:38:34.722980 18655 raft_consensus.cc:2781] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:34.723908 18655 raft_consensus.cc:686] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 1 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:38:34.725955 18753 raft_consensus.cc:482] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:34.726437 18753 raft_consensus.cc:504] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.730113 18753 leader_election.cc:288] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:34.730125 18366 tablet_bootstrap.cc:654] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:34.732028 18478 leader_election.cc:288] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:34.733203 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:38:34.734550 18249 leader_election.cc:341] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:34.739025 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:34.724771 18655 consensus_queue.cc:237] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.741804 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
W20260807 03:38:34.742113 18248 leader_election.cc:341] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:34.744225 18248 leader_election.cc:302] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:34.743588 18365 tablet_bootstrap.cc:492] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:34.744946 18753 raft_consensus.cc:2726] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:34.745024 18365 ts_tablet_manager.cc:1397] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.155s user 0.041s sys 0.061s
I20260807 03:38:34.745922 18369 tablet_bootstrap.cc:654] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
W20260807 03:38:34.747094 18027 leader_election.cc:341] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:34.743501 18191 raft_consensus.cc:2454] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:38:34.752154 18027 leader_election.cc:302] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:34.752995 18760 raft_consensus.cc:2781] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:34.753458 18760 raft_consensus.cc:686] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:38:34.753744 18365 raft_consensus.cc:348] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.754458 18365 raft_consensus.cc:374] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:34.754542 18753 raft_consensus.cc:482] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:34.754781 18365 raft_consensus.cc:729] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:34.754272 18760 consensus_queue.cc:237] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.755103 18753 raft_consensus.cc:504] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.755465 18365 consensus_queue.cc:260] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.757727 18365 ts_tablet_manager.cc:1428] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.012s user 0.006s sys 0.002s
I20260807 03:38:34.758594 18365 tablet_bootstrap.cc:492] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:34.758484 18753 leader_election.cc:288] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:34.759483 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:38:34.761520 18248 leader_election.cc:341] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:34.761669 18369 tablet_bootstrap.cc:492] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:34.762144 18369 ts_tablet_manager.cc:1397] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.080s user 0.012s sys 0.000s
I20260807 03:38:34.759490 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:34.763064 18191 raft_consensus.cc:2454] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 0.
I20260807 03:38:34.764972 18249 leader_election.cc:302] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:34.765309 18365 tablet_bootstrap.cc:654] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:34.766304 18753 raft_consensus.cc:2781] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:34.766686 18753 raft_consensus.cc:482] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:34.767087 18753 raft_consensus.cc:3037] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:34.772331 18369 raft_consensus.cc:348] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.776188 18369 raft_consensus.cc:374] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:34.776486 18369 raft_consensus.cc:729] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:34.777204 18369 consensus_queue.cc:260] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.782624 18478 consensus_queue.cc:1035] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:34.783139 18369 ts_tablet_manager.cc:1428] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20260807 03:38:34.784068 18369 tablet_bootstrap.cc:492] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:34.787098 18753 raft_consensus.cc:504] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.794718 18753 leader_election.cc:288] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:34.795321 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:34.796861 18188 raft_consensus.cc:3037] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:34.798157 17920 catalog_manager.cc:5495] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 0 to 1, leader changed from <none> to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 1 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:34.802850 18365 tablet_bootstrap.cc:492] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:34.803288 18365 ts_tablet_manager.cc:1397] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.045s user 0.013s sys 0.008s
I20260807 03:38:34.811658 18369 tablet_bootstrap.cc:654] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:34.804904 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
W20260807 03:38:34.819512 18248 leader_election.cc:341] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:34.823165 18365 raft_consensus.cc:348] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.823848 18365 raft_consensus.cc:374] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:34.824072 18366 tablet_bootstrap.cc:492] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:34.824551 18366 ts_tablet_manager.cc:1397] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.150s user 0.010s sys 0.004s
I20260807 03:38:34.824124 18365 raft_consensus.cc:729] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:34.829967 17912 catalog_manager.cc:5495] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 0 to 1, leader changed from <none> to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 1 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:34.834846 18365 consensus_queue.cc:260] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.836289 18188 raft_consensus.cc:2454] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:34.838201 18632 raft_consensus.cc:482] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:34.838706 18632 raft_consensus.cc:504] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.839557 18249 leader_election.cc:302] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:34.841207 18632 leader_election.cc:288] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:34.841428 18753 raft_consensus.cc:482] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:34.841864 18753 raft_consensus.cc:504] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.842931 18365 ts_tablet_manager.cc:1428] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.039s user 0.005s sys 0.001s
I20260807 03:38:34.843837 18365 tablet_bootstrap.cc:492] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:34.844718 18753 leader_election.cc:288] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:34.845242 18753 raft_consensus.cc:2781] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:34.845671 18753 raft_consensus.cc:686] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:34.847031 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:34.846922 18366 raft_consensus.cc:348] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.847640 18366 raft_consensus.cc:374] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:34.847909 18366 raft_consensus.cc:729] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
W20260807 03:38:34.848327 18140 leader_election.cc:341] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:34.848543 18366 consensus_queue.cc:260] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.849089 18753 consensus_queue.cc:237] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.850536 18366 ts_tablet_manager.cc:1428] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.026s user 0.004s sys 0.000s
I20260807 03:38:34.857956 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:34.859635 18188 raft_consensus.cc:2454] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 0.
I20260807 03:38:34.860909 18249 leader_election.cc:302] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:34.861755 18638 raft_consensus.cc:2781] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:34.861922 18369 tablet_bootstrap.cc:492] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:34.862116 18638 raft_consensus.cc:482] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:34.862411 18369 ts_tablet_manager.cc:1397] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.079s user 0.008s sys 0.007s
I20260807 03:38:34.862509 18365 tablet_bootstrap.cc:654] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:34.862581 18638 raft_consensus.cc:3037] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:34.864876 18369 raft_consensus.cc:348] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.865532 18369 raft_consensus.cc:374] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:34.865780 18369 raft_consensus.cc:729] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:34.866353 18369 consensus_queue.cc:260] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.868299 18369 ts_tablet_manager.cc:1428] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260807 03:38:34.869036 18369 tablet_bootstrap.cc:492] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:34.874470 18365 tablet_bootstrap.cc:492] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:34.876518 18365 ts_tablet_manager.cc:1397] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.033s user 0.012s sys 0.002s
I20260807 03:38:34.880736 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:34.882481 18365 raft_consensus.cc:348] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.883921 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:34.883037 18366 tablet_bootstrap.cc:492] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:34.884904 18079 raft_consensus.cc:3037] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
W20260807 03:38:34.885126 18248 leader_election.cc:341] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:34.886257 18369 tablet_bootstrap.cc:654] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:34.886489 18655 consensus_queue.cc:1035] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:34.888307 18623 raft_consensus.cc:482] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:34.884446 18365 raft_consensus.cc:374] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:34.888836 18623 raft_consensus.cc:504] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.889827 18365 raft_consensus.cc:729] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:34.891239 18623 leader_election.cc:288] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
W20260807 03:38:34.892907 18141 leader_election.cc:341] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:34.893375 18141 leader_election.cc:302] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:34.893503 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:34.894500 18771 raft_consensus.cc:2726] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260807 03:38:34.895633 18248 leader_election.cc:341] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:34.897100 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:38:34.898401 18249 leader_election.cc:341] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:34.898846 18249 leader_election.cc:302] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:34.899968 18623 raft_consensus.cc:2726] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:34.909199 18366 tablet_bootstrap.cc:654] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:34.909736 18365 consensus_queue.cc:260] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.913710 18638 raft_consensus.cc:504] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.928604 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
W20260807 03:38:34.930213 18248 leader_election.cc:341] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:34.938077 18369 tablet_bootstrap.cc:492] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:34.938503 18369 ts_tablet_manager.cc:1397] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.070s user 0.001s sys 0.013s
I20260807 03:38:34.939666 18365 ts_tablet_manager.cc:1428] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.062s user 0.006s sys 0.000s
I20260807 03:38:34.940589 18365 tablet_bootstrap.cc:492] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:34.951879 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:34.952581 18188 raft_consensus.cc:3037] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:34.947470 18638 leader_election.cc:288] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:34.953791 18369 raft_consensus.cc:348] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.961019 18369 raft_consensus.cc:374] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:34.960351 17912 catalog_manager.cc:5495] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 1 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:38:34.961244 18369 raft_consensus.cc:729] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:34.961849 18369 consensus_queue.cc:260] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:34.967198 18188 raft_consensus.cc:2454] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:34.968327 18249 leader_election.cc:302] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:34.989051 18756 raft_consensus.cc:2781] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:34.990461 18756 raft_consensus.cc:686] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:34.991562 18756 consensus_queue.cc:237] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.002175 18365 tablet_bootstrap.cc:654] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:35.020363 18369 ts_tablet_manager.cc:1428] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.082s user 0.007s sys 0.005s
I20260807 03:38:35.021296 18369 tablet_bootstrap.cc:492] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:35.020484 17915 catalog_manager.cc:5495] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 1 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:38:35.028580 18365 tablet_bootstrap.cc:492] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:35.029112 18365 ts_tablet_manager.cc:1397] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.089s user 0.012s sys 0.005s
I20260807 03:38:35.032238 18365 raft_consensus.cc:348] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.033146 18365 raft_consensus.cc:374] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:35.033473 18365 raft_consensus.cc:729] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:35.034610 18365 consensus_queue.cc:260] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.041281 18365 ts_tablet_manager.cc:1428] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
W20260807 03:38:35.044222 18003 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 03:38:35.052768 18365 tablet_bootstrap.cc:492] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:35.058643 18369 tablet_bootstrap.cc:654] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:35.058749 18365 tablet_bootstrap.cc:654] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:35.062212 18366 tablet_bootstrap.cc:492] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:35.062764 18366 ts_tablet_manager.cc:1397] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.180s user 0.019s sys 0.031s
I20260807 03:38:35.065127 18366 raft_consensus.cc:348] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.065827 18366 raft_consensus.cc:374] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:35.066078 18366 raft_consensus.cc:729] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:35.066493 18365 tablet_bootstrap.cc:492] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:35.066748 18366 consensus_queue.cc:260] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.068816 18365 ts_tablet_manager.cc:1397] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.004s
I20260807 03:38:35.071358 18365 raft_consensus.cc:348] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.072069 18365 raft_consensus.cc:374] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:35.072388 18365 raft_consensus.cc:729] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:35.073134 18365 consensus_queue.cc:260] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.075430 18365 ts_tablet_manager.cc:1428] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260807 03:38:35.076673 18365 tablet_bootstrap.cc:492] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:35.077958 18366 ts_tablet_manager.cc:1428] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20260807 03:38:35.081954 18365 tablet_bootstrap.cc:654] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:35.089252 18369 tablet_bootstrap.cc:492] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:35.089766 18369 ts_tablet_manager.cc:1397] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.069s user 0.016s sys 0.002s
I20260807 03:38:35.101217 18369 raft_consensus.cc:348] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.102097 18369 raft_consensus.cc:374] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:35.102421 18369 raft_consensus.cc:729] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:35.103168 18369 consensus_queue.cc:260] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.107417 18638 raft_consensus.cc:482] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:35.107918 18638 raft_consensus.cc:504] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.111523 18369 ts_tablet_manager.cc:1428] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.021s user 0.008s sys 0.000s
I20260807 03:38:35.112120 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:35.112711 18369 tablet_bootstrap.cc:492] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
W20260807 03:38:35.113404 18249 leader_election.cc:341] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:35.114187 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:38:35.115558 18248 leader_election.cc:341] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:35.115990 18248 leader_election.cc:302] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:35.116739 18753 raft_consensus.cc:2726] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:35.117482 18638 leader_election.cc:288] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:35.115737 18366 tablet_bootstrap.cc:492] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:35.133816 18366 tablet_bootstrap.cc:654] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:35.135126 18369 tablet_bootstrap.cc:654] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:35.140475 18369 tablet_bootstrap.cc:492] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:35.140918 18369 ts_tablet_manager.cc:1397] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.028s user 0.011s sys 0.000s
I20260807 03:38:35.141358 18775 raft_consensus.cc:482] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:35.141806 18365 tablet_bootstrap.cc:492] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:35.142339 18365 ts_tablet_manager.cc:1397] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.066s user 0.020s sys 0.030s
I20260807 03:38:35.141808 18775 raft_consensus.cc:504] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.143157 18369 raft_consensus.cc:348] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.143796 18369 raft_consensus.cc:374] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:35.144057 18369 raft_consensus.cc:729] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:35.144596 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:35.144719 18369 consensus_queue.cc:260] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.145452 18366 tablet_bootstrap.cc:492] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:35.145854 18366 ts_tablet_manager.cc:1397] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.003s
W20260807 03:38:35.146132 18140 leader_election.cc:341] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:35.146870 18369 ts_tablet_manager.cc:1428] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20260807 03:38:35.144743 18365 raft_consensus.cc:348] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.147817 18365 raft_consensus.cc:374] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:35.148072 18775 leader_election.cc:288] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:35.148115 18365 raft_consensus.cc:729] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:35.148568 18366 raft_consensus.cc:348] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.148753 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:35.149240 18366 raft_consensus.cc:374] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:35.149047 18365 consensus_queue.cc:260] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.149434 18312 raft_consensus.cc:2454] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 0.
I20260807 03:38:35.149537 18366 raft_consensus.cc:729] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:35.150197 18366 consensus_queue.cc:260] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.150607 18141 leader_election.cc:302] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:35.151036 18365 ts_tablet_manager.cc:1428] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260807 03:38:35.151433 18771 raft_consensus.cc:2781] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:35.151791 18369 tablet_bootstrap.cc:492] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:35.152021 18366 ts_tablet_manager.cc:1428] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260807 03:38:35.153347 18366 tablet_bootstrap.cc:492] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:35.157199 18369 tablet_bootstrap.cc:654] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:35.158463 18366 tablet_bootstrap.cc:654] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:35.161492 18365 tablet_bootstrap.cc:492] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:35.151767 18771 raft_consensus.cc:482] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:35.161962 18771 raft_consensus.cc:3037] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:35.176548 18632 raft_consensus.cc:482] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:35.177002 18632 raft_consensus.cc:504] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.178862 18632 leader_election.cc:288] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:35.180353 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:35.183156 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:35.183298 18191 raft_consensus.cc:3037] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:35.183946 18305 raft_consensus.cc:2379] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 1: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:38:35.185418 18365 tablet_bootstrap.cc:654] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:35.186522 18141 leader_election.cc:302] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:35.190047 18478 consensus_queue.cc:1035] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 03:38:35.209084 18365 tablet_bootstrap.cc:492] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:35.209553 18365 ts_tablet_manager.cc:1397] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.048s user 0.013s sys 0.005s
I20260807 03:38:35.212097 18365 raft_consensus.cc:348] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.212818 18365 raft_consensus.cc:374] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:35.213065 18365 raft_consensus.cc:729] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:35.213721 18365 consensus_queue.cc:260] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.224231 18632 raft_consensus.cc:2726] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:35.227720 18478 consensus_queue.cc:1035] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 03:38:35.230237 18365 ts_tablet_manager.cc:1428] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.020s user 0.003s sys 0.010s
I20260807 03:38:35.231132 18365 tablet_bootstrap.cc:492] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:35.233227 18305 raft_consensus.cc:3037] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:35.247604 18771 raft_consensus.cc:504] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.249691 18771 leader_election.cc:288] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:35.241503 18756 raft_consensus.cc:482] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:35.251686 18756 raft_consensus.cc:504] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.253948 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:35.254426 18756 leader_election.cc:288] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
W20260807 03:38:35.256237 18140 leader_election.cc:341] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:35.257587 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:35.258195 18193 raft_consensus.cc:2454] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 0.
I20260807 03:38:35.258249 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:35.260339 18249 leader_election.cc:302] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:35.260675 18655 consensus_queue.cc:1035] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 03:38:35.262163 18248 leader_election.cc:341] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:35.263150 18638 raft_consensus.cc:2781] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:35.263434 18365 tablet_bootstrap.cc:654] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:35.263553 18638 raft_consensus.cc:482] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:35.263929 18638 raft_consensus.cc:3037] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:35.274595 18638 raft_consensus.cc:504] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.281864 18753 raft_consensus.cc:482] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 03:38:35.288268 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 74934.5us
I20260807 03:38:35.291973 18753 raft_consensus.cc:504] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.294135 18753 leader_election.cc:288] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:35.294291 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:35.294845 18309 raft_consensus.cc:3037] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:35.289343 18366 tablet_bootstrap.cc:492] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:35.297250 18366 ts_tablet_manager.cc:1397] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.144s user 0.012s sys 0.035s
I20260807 03:38:35.302940 18478 consensus_queue.cc:1035] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260807 03:38:35.302884 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:35.305776 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:35.309181 18248 leader_election.cc:302] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:35.311685 18309 raft_consensus.cc:2454] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:35.313416 18141 leader_election.cc:302] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:35.314785 18366 raft_consensus.cc:348] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.315623 18366 raft_consensus.cc:374] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:35.318194 18638 leader_election.cc:288] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:35.319634 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:35.320282 18188 raft_consensus.cc:3037] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:35.323658 18365 tablet_bootstrap.cc:492] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:35.324563 18365 ts_tablet_manager.cc:1397] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.094s user 0.009s sys 0.012s
I20260807 03:38:35.323669 18369 tablet_bootstrap.cc:492] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:35.326417 18369 ts_tablet_manager.cc:1397] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.175s user 0.016s sys 0.043s
I20260807 03:38:35.315949 18366 raft_consensus.cc:729] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:35.329079 18366 consensus_queue.cc:260] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.332083 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:35.342196 18884 raft_consensus.cc:2781] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:35.342664 18884 raft_consensus.cc:686] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:35.343511 18884 consensus_queue.cc:237] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:35.347271 18248 leader_election.cc:341] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:35.348161 18756 raft_consensus.cc:2726] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:35.354475 18757 consensus_queue.cc:1035] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:35.361842 18080 raft_consensus.cc:3037] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:35.362254 18369 raft_consensus.cc:348] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.362962 18369 raft_consensus.cc:374] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:35.363240 18369 raft_consensus.cc:729] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:35.363826 18369 consensus_queue.cc:260] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.366880 18366 ts_tablet_manager.cc:1428] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.069s user 0.007s sys 0.000s
I20260807 03:38:35.372474 18369 ts_tablet_manager.cc:1428] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.046s user 0.004s sys 0.003s
I20260807 03:38:35.373798 18366 tablet_bootstrap.cc:492] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:35.374917 18369 tablet_bootstrap.cc:492] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:35.379199 18365 raft_consensus.cc:348] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.380420 18365 raft_consensus.cc:374] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:35.380760 18365 raft_consensus.cc:729] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:35.381584 18365 consensus_queue.cc:260] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.392503 18365 ts_tablet_manager.cc:1428] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.066s user 0.004s sys 0.004s
I20260807 03:38:35.395318 18365 tablet_bootstrap.cc:492] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:35.407135 18756 raft_consensus.cc:482] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:35.407655 18756 raft_consensus.cc:504] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.408588 18478 raft_consensus.cc:482] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:35.409081 18478 raft_consensus.cc:504] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.409595 18756 leader_election.cc:288] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:35.410195 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:35.411017 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:38:35.411525 18248 leader_election.cc:341] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:35.411628 18193 raft_consensus.cc:2454] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 0.
I20260807 03:38:35.412343 18188 raft_consensus.cc:2454] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:35.412899 18249 leader_election.cc:302] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:35.412567 18478 leader_election.cc:288] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:35.416277 18249 leader_election.cc:302] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:35.417616 18757 raft_consensus.cc:2781] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:35.415956 18756 raft_consensus.cc:2781] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:35.418030 18638 consensus_queue.cc:1035] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:35.418443 18756 raft_consensus.cc:482] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:35.418754 18756 raft_consensus.cc:3037] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:35.418054 18757 raft_consensus.cc:686] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:35.420039 18757 consensus_queue.cc:237] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.427522 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:35.432428 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:35.435005 17912 catalog_manager.cc:5495] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 1 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:35.446300 18897 raft_consensus.cc:482] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:35.454937 18369 tablet_bootstrap.cc:654] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:35.457018 18884 raft_consensus.cc:482] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:35.457644 18884 raft_consensus.cc:504] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.460203 18884 leader_election.cc:288] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:35.461515 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:35.462987 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:38:35.463272 18140 leader_election.cc:341] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:35.463598 18309 raft_consensus.cc:2454] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 0.
I20260807 03:38:35.464699 18141 leader_election.cc:302] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:35.456374 18897 raft_consensus.cc:504] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.467636 18080 raft_consensus.cc:3037] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:35.448055 18756 raft_consensus.cc:504] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.470973 18883 raft_consensus.cc:482] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:35.492278 18883 raft_consensus.cc:504] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.494946 18883 leader_election.cc:288] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:35.469908 18884 raft_consensus.cc:2781] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:35.495589 18884 raft_consensus.cc:482] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:35.495988 18884 raft_consensus.cc:3037] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:35.496114 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:35.496177 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:35.496830 18309 raft_consensus.cc:2454] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 0.
I20260807 03:38:35.496982 18073 raft_consensus.cc:2454] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 0.
I20260807 03:38:35.498014 18141 leader_election.cc:302] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:35.499109 18883 raft_consensus.cc:2781] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:35.499491 18883 raft_consensus.cc:482] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:35.480378 18771 raft_consensus.cc:482] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:35.499856 18883 raft_consensus.cc:3037] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:35.481160 18897 leader_election.cc:288] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:35.483253 18366 tablet_bootstrap.cc:654] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:35.485598 18365 tablet_bootstrap.cc:654] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:35.487075 18027 leader_election.cc:302] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:35.500420 18771 raft_consensus.cc:504] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.514166 18369 tablet_bootstrap.cc:492] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:35.514627 18369 ts_tablet_manager.cc:1397] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.140s user 0.006s sys 0.006s
I20260807 03:38:35.519214 17912 catalog_manager.cc:5495] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 1 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:38:35.524775 18756 leader_election.cc:288] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:35.525492 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:35.526652 18884 raft_consensus.cc:504] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.529044 18884 leader_election.cc:288] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:35.531147 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:35.514165 18771 leader_election.cc:288] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:35.535036 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:35.535468 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:35.538177 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:35.538820 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:35.532235 18369 raft_consensus.cc:348] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.542693 18369 raft_consensus.cc:374] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:35.543074 18369 raft_consensus.cc:729] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
W20260807 03:38:35.540585 18140 leader_election.cc:341] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:35.543865 18369 consensus_queue.cc:260] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.540671 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:35.547969 18188 raft_consensus.cc:3037] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:35.554924 18655 raft_consensus.cc:482] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:35.555354 18655 raft_consensus.cc:504] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:35.556298 18140 leader_election.cc:341] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:35.557842 18655 leader_election.cc:288] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:35.544759 18305 raft_consensus.cc:3037] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:35.561373 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:35.562049 18191 raft_consensus.cc:2454] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 0.
I20260807 03:38:35.561100 18193 raft_consensus.cc:2379] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 1: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:38:35.563082 18369 ts_tablet_manager.cc:1428] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.048s user 0.009s sys 0.000s
I20260807 03:38:35.563405 18027 leader_election.cc:302] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:35.566882 18369 tablet_bootstrap.cc:492] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:35.560561 18753 consensus_queue.cc:1035] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260807 03:38:35.567956 18027 leader_election.cc:302] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:35.568727 18760 raft_consensus.cc:2726] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:35.547346 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:35.566395 18655 raft_consensus.cc:2781] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260807 03:38:35.577692 18248 leader_election.cc:341] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:35.577807 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:35.572970 18897 raft_consensus.cc:2726] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:35.578339 18312 raft_consensus.cc:2454] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 0.
W20260807 03:38:35.584842 18141 leader_election.cc:341] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:35.585307 18141 leader_election.cc:302] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:35.577198 18655 raft_consensus.cc:482] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:35.589427 18775 raft_consensus.cc:2726] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:35.598179 18655 raft_consensus.cc:3037] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:35.592331 18305 raft_consensus.cc:2454] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:35.621181 18623 consensus_queue.cc:1035] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 03:38:35.625981 18188 raft_consensus.cc:2454] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:35.629909 18249 leader_election.cc:302] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:35.632020 18757 raft_consensus.cc:2781] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:35.633188 18655 raft_consensus.cc:504] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.635834 18757 raft_consensus.cc:686] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:35.641630 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:35.642215 18193 raft_consensus.cc:3037] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:35.645200 18141 leader_election.cc:302] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:35.645411 18655 leader_election.cc:288] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:35.645593 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:35.646313 18311 raft_consensus.cc:3037] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:35.638720 18757 consensus_queue.cc:237] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.653028 18193 raft_consensus.cc:2454] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:38:35.653394 18311 raft_consensus.cc:2454] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:38:35.654062 18027 leader_election.cc:302] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:35.656648 18771 raft_consensus.cc:2781] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election won for term 1
W20260807 03:38:35.657471 18901 log.cc:924] Time spent T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.080s user 0.001s sys 0.000s
I20260807 03:38:35.661088 18369 tablet_bootstrap.cc:654] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:35.661834 18771 raft_consensus.cc:686] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:35.661337 18655 raft_consensus.cc:2781] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:35.664568 18771 consensus_queue.cc:237] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.665427 18366 tablet_bootstrap.cc:492] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:35.665877 18366 ts_tablet_manager.cc:1397] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.296s user 0.021s sys 0.005s
I20260807 03:38:35.665915 18655 raft_consensus.cc:686] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:38:35.666981 18655 consensus_queue.cc:237] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.668316 18366 raft_consensus.cc:348] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.669014 18366 raft_consensus.cc:374] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:35.669278 18366 raft_consensus.cc:729] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:35.669868 18366 consensus_queue.cc:260] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.673213 18366 ts_tablet_manager.cc:1428] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260807 03:38:35.674258 18366 tablet_bootstrap.cc:492] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:35.660768 18883 raft_consensus.cc:504] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.697535 18883 leader_election.cc:288] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:35.698256 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:35.698483 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:35.699015 18075 raft_consensus.cc:3037] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:35.699229 18311 raft_consensus.cc:3037] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:35.707335 18366 tablet_bootstrap.cc:654] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:35.709710 18365 tablet_bootstrap.cc:492] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:35.709998 18311 raft_consensus.cc:2454] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:35.710282 18365 ts_tablet_manager.cc:1397] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.317s user 0.020s sys 0.012s
I20260807 03:38:35.711272 18141 leader_election.cc:302] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:35.714037 18075 raft_consensus.cc:2454] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:35.715986 18775 raft_consensus.cc:2781] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:35.719395 18365 raft_consensus.cc:348] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.730163 18365 raft_consensus.cc:374] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:35.730403 18365 raft_consensus.cc:729] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:35.731123 18365 consensus_queue.cc:260] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.733150 18775 raft_consensus.cc:686] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:35.732717 17914 catalog_manager.cc:5495] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 0 to 1, leader changed from <none> to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 1 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:35.738253 18775 consensus_queue.cc:237] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.738579 18365 ts_tablet_manager.cc:1428] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.028s user 0.003s sys 0.002s
I20260807 03:38:35.740276 18365 tablet_bootstrap.cc:492] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:35.763780 18365 tablet_bootstrap.cc:654] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:35.766057 18369 tablet_bootstrap.cc:492] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:35.766557 18369 ts_tablet_manager.cc:1397] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.203s user 0.014s sys 0.008s
I20260807 03:38:35.769091 18369 raft_consensus.cc:348] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.769757 18369 raft_consensus.cc:374] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:35.769984 18369 raft_consensus.cc:729] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:35.770503 18369 consensus_queue.cc:260] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.771971 18369 ts_tablet_manager.cc:1428] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260807 03:38:35.773289 18369 tablet_bootstrap.cc:492] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:35.777894 18366 tablet_bootstrap.cc:492] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:35.778311 18366 ts_tablet_manager.cc:1397] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.104s user 0.007s sys 0.005s
I20260807 03:38:35.780617 18366 raft_consensus.cc:348] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.781260 18366 raft_consensus.cc:374] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:35.781597 18366 raft_consensus.cc:729] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:35.782312 18366 consensus_queue.cc:260] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.784302 18366 ts_tablet_manager.cc:1428] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260807 03:38:35.784328 17912 catalog_manager.cc:5495] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 1 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:38:35.785183 18366 tablet_bootstrap.cc:492] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:35.790139 18081 raft_consensus.cc:3037] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:35.790956 18369 tablet_bootstrap.cc:654] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:35.794533 17920 catalog_manager.cc:5495] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 1 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:35.806464 18366 tablet_bootstrap.cc:654] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:35.809648 18775 consensus_queue.cc:1035] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:35.812837 18365 tablet_bootstrap.cc:492] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:35.813278 18365 ts_tablet_manager.cc:1397] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.073s user 0.010s sys 0.011s
I20260807 03:38:35.816457 18365 raft_consensus.cc:348] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.817243 18365 raft_consensus.cc:374] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:35.817576 18365 raft_consensus.cc:729] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:35.818336 18365 consensus_queue.cc:260] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.822266 18369 tablet_bootstrap.cc:492] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:35.822871 18369 ts_tablet_manager.cc:1397] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.050s user 0.007s sys 0.001s
I20260807 03:38:35.826095 18369 raft_consensus.cc:348] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.826740 18369 raft_consensus.cc:374] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:35.827001 18369 raft_consensus.cc:729] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:35.827726 18369 consensus_queue.cc:260] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.829731 18369 ts_tablet_manager.cc:1428] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s
I20260807 03:38:35.830540 18369 tablet_bootstrap.cc:492] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:35.836069 18369 tablet_bootstrap.cc:654] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:35.838042 17920 catalog_manager.cc:5495] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 1 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:35.842252 18366 tablet_bootstrap.cc:492] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:35.842669 18366 ts_tablet_manager.cc:1397] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.058s user 0.013s sys 0.022s
I20260807 03:38:35.845103 18366 raft_consensus.cc:348] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.845731 18366 raft_consensus.cc:374] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:35.845952 18366 raft_consensus.cc:729] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:35.846552 18366 consensus_queue.cc:260] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.848712 18366 ts_tablet_manager.cc:1428] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260807 03:38:35.848958 18365 ts_tablet_manager.cc:1428] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.035s user 0.006s sys 0.002s
I20260807 03:38:35.849462 18366 tablet_bootstrap.cc:492] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:35.849852 18365 tablet_bootstrap.cc:492] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:35.855394 18366 tablet_bootstrap.cc:654] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:35.864861 18369 tablet_bootstrap.cc:492] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:35.865271 18369 ts_tablet_manager.cc:1397] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.035s user 0.005s sys 0.004s
I20260807 03:38:35.866397 18771 consensus_queue.cc:1035] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:35.867597 18369 raft_consensus.cc:348] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.868274 18369 raft_consensus.cc:374] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:35.868571 18369 raft_consensus.cc:729] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:35.869158 18369 consensus_queue.cc:260] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.871125 18623 raft_consensus.cc:482] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:35.871560 18369 ts_tablet_manager.cc:1428] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260807 03:38:35.863317 18365 tablet_bootstrap.cc:654] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:35.871682 18623 raft_consensus.cc:504] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.874414 18623 leader_election.cc:288] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:35.877230 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:35.878688 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:38:35.880446 18249 leader_election.cc:341] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:35.881017 18369 tablet_bootstrap.cc:492] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
W20260807 03:38:35.881908 18248 leader_election.cc:341] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:35.882454 18248 leader_election.cc:302] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:35.886740 18753 raft_consensus.cc:2726] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:35.892307 18365 tablet_bootstrap.cc:492] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:35.892735 18365 ts_tablet_manager.cc:1397] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.043s user 0.010s sys 0.001s
I20260807 03:38:35.895049 18365 raft_consensus.cc:348] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.895685 18365 raft_consensus.cc:374] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:35.895934 18365 raft_consensus.cc:729] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:35.899570 18369 tablet_bootstrap.cc:654] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:35.900375 18365 consensus_queue.cc:260] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:35.907160 18248 consensus_peers.cc:487] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 -> Peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Couldn't send request to peer d9716e2bf06442f5b3b0e81087c53c4d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260807 03:38:35.909346 18366 tablet_bootstrap.cc:492] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:35.909795 18366 ts_tablet_manager.cc:1397] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.060s user 0.023s sys 0.024s
I20260807 03:38:35.910125 18365 ts_tablet_manager.cc:1428] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20260807 03:38:35.910985 18365 tablet_bootstrap.cc:492] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:35.917934 18365 tablet_bootstrap.cc:654] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:35.912065 18366 raft_consensus.cc:348] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.921584 18366 raft_consensus.cc:374] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:35.921854 18366 raft_consensus.cc:729] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:35.922513 18366 consensus_queue.cc:260] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.928133 18369 tablet_bootstrap.cc:492] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:35.928571 18369 ts_tablet_manager.cc:1397] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.048s user 0.010s sys 0.001s
I20260807 03:38:35.932060 18369 raft_consensus.cc:348] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.932595 18366 ts_tablet_manager.cc:1428] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.023s user 0.007s sys 0.000s
I20260807 03:38:35.932756 18369 raft_consensus.cc:374] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:35.933045 18369 raft_consensus.cc:729] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:35.933611 18369 consensus_queue.cc:260] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.934624 18366 tablet_bootstrap.cc:492] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:35.935763 18369 ts_tablet_manager.cc:1428] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260807 03:38:35.937376 18369 tablet_bootstrap.cc:492] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:35.942852 18366 tablet_bootstrap.cc:654] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:35.943849 18369 tablet_bootstrap.cc:654] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:35.959187 18753 consensus_queue.cc:1035] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:35.962463 18365 tablet_bootstrap.cc:492] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:35.962883 18365 ts_tablet_manager.cc:1397] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.052s user 0.008s sys 0.008s
I20260807 03:38:35.965786 18365 raft_consensus.cc:348] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.967278 18365 raft_consensus.cc:374] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:35.967634 18365 raft_consensus.cc:729] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:35.968447 18366 tablet_bootstrap.cc:492] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:35.968538 18771 raft_consensus.cc:482] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:35.968952 18366 ts_tablet_manager.cc:1397] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.034s user 0.014s sys 0.005s
I20260807 03:38:35.969058 18771 raft_consensus.cc:504] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.969828 18365 consensus_queue.cc:260] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.971436 18771 leader_election.cc:288] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:35.971590 18366 raft_consensus.cc:348] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.972211 18366 raft_consensus.cc:374] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:35.972484 18366 raft_consensus.cc:729] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:35.973055 18366 consensus_queue.cc:260] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.973379 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:35.974082 18311 raft_consensus.cc:2454] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 0.
I20260807 03:38:35.975334 18366 ts_tablet_manager.cc:1428] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20260807 03:38:35.976078 18366 tablet_bootstrap.cc:492] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:35.977244 18141 leader_election.cc:302] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:35.976799 18369 tablet_bootstrap.cc:492] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:35.973078 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:35.978623 18369 ts_tablet_manager.cc:1397] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.041s user 0.014s sys 0.005s
W20260807 03:38:35.979702 18140 leader_election.cc:341] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:35.978035 18771 raft_consensus.cc:2781] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:35.982620 18771 raft_consensus.cc:482] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:35.982998 18771 raft_consensus.cc:3037] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:35.984202 18369 raft_consensus.cc:348] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.985064 18369 raft_consensus.cc:374] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:35.985364 18369 raft_consensus.cc:729] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:35.986029 18369 consensus_queue.cc:260] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.988782 18369 ts_tablet_manager.cc:1428] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.010s user 0.006s sys 0.001s
I20260807 03:38:35.990640 18369 tablet_bootstrap.cc:492] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:35.989882 18623 raft_consensus.cc:482] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:35.991709 18623 raft_consensus.cc:504] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:35.993804 18623 leader_election.cc:288] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:35.996963 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:35.997538 18193 raft_consensus.cc:2454] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 0.
I20260807 03:38:35.998214 18366 tablet_bootstrap.cc:654] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:35.996991 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:36.000983 18249 leader_election.cc:302] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
W20260807 03:38:36.002713 18248 leader_election.cc:341] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:36.002892 18623 raft_consensus.cc:2781] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:36.003254 18623 raft_consensus.cc:482] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:36.004040 18365 ts_tablet_manager.cc:1428] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.041s user 0.006s sys 0.002s
I20260807 03:38:36.003923 18623 raft_consensus.cc:3037] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:36.006214 18365 tablet_bootstrap.cc:492] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:36.008798 18771 raft_consensus.cc:504] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:36.016285 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:36.016868 18369 tablet_bootstrap.cc:654] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
W20260807 03:38:36.017848 18140 leader_election.cc:341] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:36.019830 18881 raft_consensus.cc:482] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:36.020495 18365 tablet_bootstrap.cc:654] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:36.020248 18881 raft_consensus.cc:504] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:36.021709 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:36.022563 18305 raft_consensus.cc:3037] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:36.025708 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:38:36.027190 18141 leader_election.cc:341] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:36.023293 18881 leader_election.cc:288] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:36.035822 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:38:36.037756 18140 leader_election.cc:341] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:36.038296 18140 leader_election.cc:302] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:36.039192 18775 raft_consensus.cc:2726] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:36.040447 18753 raft_consensus.cc:482] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:36.041070 18753 raft_consensus.cc:504] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:36.022011 18771 leader_election.cc:288] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:36.043488 18753 leader_election.cc:288] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:36.056456 18775 raft_consensus.cc:482] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:36.043982 18757 raft_consensus.cc:482] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:36.045740 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:36.056926 18775 raft_consensus.cc:504] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:36.057125 18757 raft_consensus.cc:504] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:36.058264 18249 leader_election.cc:341] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:36.058787 18775 leader_election.cc:288] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:36.060772 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:36.061105 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:36.061386 18311 raft_consensus.cc:2454] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 0.
I20260807 03:38:36.060755 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:38:36.062197 18248 leader_election.cc:341] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:36.062929 18141 leader_election.cc:302] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
W20260807 03:38:36.063344 18249 leader_election.cc:341] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:36.063776 18249 leader_election.cc:302] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:36.064503 18756 raft_consensus.cc:2726] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:36.065114 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:36.065757 18081 raft_consensus.cc:2454] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 0.
I20260807 03:38:36.063802 18775 raft_consensus.cc:2781] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:36.074692 18248 leader_election.cc:302] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:38:36.074955 18775 raft_consensus.cc:482] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:36.075454 18775 raft_consensus.cc:3037] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:36.076857 18756 raft_consensus.cc:2781] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:36.077215 18756 raft_consensus.cc:482] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:36.077455 18897 raft_consensus.cc:482] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:36.078223 18756 raft_consensus.cc:3037] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:36.079118 18897 raft_consensus.cc:504] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:36.081061 18897 leader_election.cc:288] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:36.059852 18757 leader_election.cc:288] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:36.083616 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:36.084482 18081 raft_consensus.cc:2454] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 0.
I20260807 03:38:36.086787 18623 raft_consensus.cc:504] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:36.089982 18897 consensus_queue.cc:1035] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 03:38:36.090683 18623 leader_election.cc:288] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:36.094128 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:36.094877 18312 raft_consensus.cc:2454] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 0.
I20260807 03:38:36.100674 18366 tablet_bootstrap.cc:492] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:36.100674 18365 tablet_bootstrap.cc:492] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:36.101229 18366 ts_tablet_manager.cc:1397] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.125s user 0.025s sys 0.017s
I20260807 03:38:36.101229 18365 ts_tablet_manager.cc:1397] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.095s user 0.033s sys 0.025s
I20260807 03:38:36.101827 18027 leader_election.cc:302] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:36.121052 18366 raft_consensus.cc:348] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:36.121587 18366 raft_consensus.cc:374] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:36.086591 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:36.121855 18366 raft_consensus.cc:729] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:36.122541 18366 consensus_queue.cc:260] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:36.103770 18897 raft_consensus.cc:2781] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:36.103883 18365 raft_consensus.cc:348] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:36.124713 18365 raft_consensus.cc:374] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:36.125078 18365 raft_consensus.cc:729] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:36.125705 18365 consensus_queue.cc:260] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:36.107422 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:36.148062 18365 ts_tablet_manager.cc:1428] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.046s user 0.006s sys 0.000s
I20260807 03:38:36.149199 18365 tablet_bootstrap.cc:492] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:36.150635 18366 ts_tablet_manager.cc:1428] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.049s user 0.003s sys 0.004s
I20260807 03:38:36.153990 18366 tablet_bootstrap.cc:492] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:36.124569 18897 raft_consensus.cc:482] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 03:38:36.158953 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 61066.3us
W20260807 03:38:36.165099 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 83523.2us
I20260807 03:38:36.161661 18897 raft_consensus.cc:3037] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:36.134135 18756 raft_consensus.cc:504] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:36.178368 18369 tablet_bootstrap.cc:492] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:36.179415 18756 leader_election.cc:288] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:36.181813 18775 raft_consensus.cc:504] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:36.181928 18897 raft_consensus.cc:504] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:36.183277 18775 leader_election.cc:288] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:36.183374 18897 leader_election.cc:288] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:38:36.220350 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 82920.7us
I20260807 03:38:36.234701 18189 raft_consensus.cc:2379] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 1: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:38:36.237416 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:36.238179 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:36.238967 18193 raft_consensus.cc:3037] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:36.239217 18082 raft_consensus.cc:3037] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:36.254063 18369 ts_tablet_manager.cc:1397] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.263s user 0.008s sys 0.006s
I20260807 03:38:36.257084 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:36.257876 18311 raft_consensus.cc:3037] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:36.264451 18760 raft_consensus.cc:482] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:36.265268 18760 raft_consensus.cc:504] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:36.281687 18760 leader_election.cc:288] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:38:36.290617 18003 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 03:38:36.291126 18365 tablet_bootstrap.cc:654] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:36.293411 18369 raft_consensus.cc:348] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:36.294405 18369 raft_consensus.cc:374] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:36.294736 18369 raft_consensus.cc:729] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:36.294756 18366 tablet_bootstrap.cc:654] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:36.311568 18082 raft_consensus.cc:2454] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:36.295608 18369 consensus_queue.cc:260] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:36.274849 18248 leader_election.cc:341] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:36.296831 18305 raft_consensus.cc:2454] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
W20260807 03:38:36.350983 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 113855us
I20260807 03:38:36.351727 18311 raft_consensus.cc:2454] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:38:36.353215 18897 raft_consensus.cc:482] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:36.365803 18193 raft_consensus.cc:2454] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:36.354979 18760 raft_consensus.cc:482] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:36.375763 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:36.378674 18760 raft_consensus.cc:504] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:36.383061 18369 ts_tablet_manager.cc:1428] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.128s user 0.005s sys 0.012s
I20260807 03:38:36.383812 18027 leader_election.cc:302] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:36.384795 18760 leader_election.cc:288] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:36.385380 18961 raft_consensus.cc:2781] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:36.386205 18961 raft_consensus.cc:686] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 1 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:38:36.391053 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:36.394811 18961 consensus_queue.cc:237] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:36.395118 18897 raft_consensus.cc:504] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:36.376698 18189 raft_consensus.cc:3037] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:36.406513 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:36.411779 18365 tablet_bootstrap.cc:492] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:36.412580 18365 ts_tablet_manager.cc:1397] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.264s user 0.012s sys 0.012s
I20260807 03:38:36.415014 18081 raft_consensus.cc:3037] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:36.417718 18311 raft_consensus.cc:2454] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 0.
I20260807 03:38:36.416296 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
W20260807 03:38:36.424371 18248 consensus_peers.cc:487] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 -> Peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Couldn't send request to peer d9716e2bf06442f5b3b0e81087c53c4d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260807 03:38:36.425625 18248 leader_election.cc:302] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:36.427450 18753 raft_consensus.cc:2781] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:36.428045 18753 raft_consensus.cc:686] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:36.429258 18753 consensus_queue.cc:237] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:36.456532 18027 leader_election.cc:302] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:36.454172 18249 leader_election.cc:302] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:36.459745 18964 raft_consensus.cc:2781] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:36.460345 18964 raft_consensus.cc:482] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:36.460800 18964 raft_consensus.cc:3037] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:36.457748 18141 leader_election.cc:302] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:36.466500 18771 raft_consensus.cc:482] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:36.467159 18771 raft_consensus.cc:504] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:36.469218 18369 tablet_bootstrap.cc:492] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:36.469381 18771 leader_election.cc:288] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:36.472596 18775 raft_consensus.cc:2781] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:36.473116 18775 raft_consensus.cc:686] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:36.473992 18775 consensus_queue.cc:237] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:36.440227 18897 leader_election.cc:288] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:36.494098 18623 raft_consensus.cc:2781] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:36.494871 18623 raft_consensus.cc:686] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
W20260807 03:38:36.496632 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 93122.1us
I20260807 03:38:36.493113 18365 raft_consensus.cc:348] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:36.504707 18365 raft_consensus.cc:374] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:36.504962 18365 raft_consensus.cc:729] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:36.505492 18365 consensus_queue.cc:260] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:36.506234 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:36.508831 18310 raft_consensus.cc:2379] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 1: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:38:36.496075 18623 consensus_queue.cc:237] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:36.523608 18189 raft_consensus.cc:2454] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:36.529017 18365 ts_tablet_manager.cc:1428] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.116s user 0.005s sys 0.001s
I20260807 03:38:36.533542 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:36.536172 18193 raft_consensus.cc:2379] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 1: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:38:36.537314 18946 consensus_queue.cc:1035] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260807 03:38:36.543248 18366 tablet_bootstrap.cc:492] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:36.545116 18366 ts_tablet_manager.cc:1397] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.391s user 0.025s sys 0.040s
I20260807 03:38:36.545622 18365 tablet_bootstrap.cc:492] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:36.546088 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:36.547942 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:36.556824 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:36.557670 18305 raft_consensus.cc:2379] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 1: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:38:36.556535 18366 raft_consensus.cc:348] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:36.559535 18366 raft_consensus.cc:374] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:36.559976 18366 raft_consensus.cc:729] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:36.561014 18311 raft_consensus.cc:2454] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 0.
I20260807 03:38:36.561782 18366 consensus_queue.cc:260] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:36.646965 18081 raft_consensus.cc:3037] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:36.581525 18950 raft_consensus.cc:482] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:36.649405 18950 raft_consensus.cc:504] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:36.591647 18963 log.cc:924] Time spent T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.108s user 0.000s sys 0.001s
I20260807 03:38:36.594221 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:36.655606 18950 leader_election.cc:288] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:36.656881 18366 ts_tablet_manager.cc:1428] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.111s user 0.038s sys 0.045s
I20260807 03:38:36.649861 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:36.654420 18369 tablet_bootstrap.cc:654] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:36.672400 18883 consensus_queue.cc:1035] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260807 03:38:36.672689 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:36.673725 18190 raft_consensus.cc:2454] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 0.
I20260807 03:38:36.676077 18249 leader_election.cc:302] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:36.676952 18141 leader_election.cc:302] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:36.676501 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:36.691640 18956 raft_consensus.cc:2726] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260807 03:38:36.692394 18756 raft_consensus.cc:2781] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:36.692860 18756 raft_consensus.cc:482] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:36.693240 18756 raft_consensus.cc:3037] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:36.696370 18366 tablet_bootstrap.cc:492] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:36.698225 18073 raft_consensus.cc:2454] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 0.
I20260807 03:38:36.683475 18883 consensus_queue.cc:1035] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20260807 03:38:36.701184 17914 catalog_manager.cc:5495] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 1 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:38:36.708150 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:36.708117 17920 catalog_manager.cc:5495] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 0 to 1, leader changed from <none> to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 1 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:36.723346 18655 consensus_queue.cc:1035] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 03:38:36.725258 18027 leader_election.cc:302] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:36.726656 18365 tablet_bootstrap.cc:654] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:36.728209 17914 catalog_manager.cc:5495] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 1 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:38:36.735257 18964 raft_consensus.cc:504] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:36.741660 18027 leader_election.cc:341] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:36.743276 18964 leader_election.cc:288] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:36.750046 18141 leader_election.cc:302] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:36.750833 18027 leader_election.cc:302] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:36.752988 18655 raft_consensus.cc:2726] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:36.768987 18960 raft_consensus.cc:2781] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:36.769402 18960 raft_consensus.cc:482] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:36.769749 18960 raft_consensus.cc:3037] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:36.764254 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:36.774580 18305 raft_consensus.cc:3037] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:36.775902 18949 raft_consensus.cc:2726] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:36.781418 18366 tablet_bootstrap.cc:654] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:36.838176 18960 raft_consensus.cc:504] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:36.849505 18982 log.cc:924] Time spent T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
I20260807 03:38:36.776959 18756 raft_consensus.cc:504] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:36.859016 18305 raft_consensus.cc:2454] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:38:36.861449 18756 leader_election.cc:288] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:36.862377 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:36.863152 18081 raft_consensus.cc:3037] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:36.859845 18369 tablet_bootstrap.cc:492] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:36.869174 18369 ts_tablet_manager.cc:1397] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.400s user 0.014s sys 0.014s
I20260807 03:38:36.870734 18027 leader_election.cc:302] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:36.872304 18655 raft_consensus.cc:2781] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:36.873546 18655 raft_consensus.cc:686] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:38:36.873920 18365 tablet_bootstrap.cc:492] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:36.874372 18365 ts_tablet_manager.cc:1397] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.329s user 0.007s sys 0.008s
I20260807 03:38:36.874444 18655 consensus_queue.cc:237] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:36.878197 18365 raft_consensus.cc:348] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:36.884338 18365 raft_consensus.cc:374] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:36.884634 18365 raft_consensus.cc:729] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:36.874467 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:36.885234 18365 consensus_queue.cc:260] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:36.888661 18081 raft_consensus.cc:2454] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:36.897295 18248 leader_election.cc:302] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:36.905890 18369 raft_consensus.cc:348] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:36.913072 18369 raft_consensus.cc:374] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:36.913445 18369 raft_consensus.cc:729] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
W20260807 03:38:36.939451 18140 leader_election.cc:341] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:36.912145 18187 raft_consensus.cc:3037] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:36.945469 18771 consensus_queue.cc:1035] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260807 03:38:36.954044 18881 consensus_queue.cc:1035] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.038s
I20260807 03:38:36.955436 18369 consensus_queue.cc:260] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:36.915932 18623 raft_consensus.cc:2781] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election won for term 1
W20260807 03:38:36.955888 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 44066.8us
I20260807 03:38:36.939980 18956 raft_consensus.cc:482] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:36.934906 18960 leader_election.cc:288] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:38:36.964221 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 45207.7us
I20260807 03:38:36.967252 18956 raft_consensus.cc:504] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:36.992533 18956 leader_election.cc:288] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:36.985913 18623 raft_consensus.cc:686] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:36.994056 18623 consensus_queue.cc:237] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:37.008208 18369 ts_tablet_manager.cc:1428] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.139s user 0.004s sys 0.005s
I20260807 03:38:37.009755 18365 ts_tablet_manager.cc:1428] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.135s user 0.009s sys 0.001s
W20260807 03:38:37.017812 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 51305.5us
W20260807 03:38:37.019948 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 73420.1us
I20260807 03:38:37.011081 18950 raft_consensus.cc:482] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:37.021967 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:37.022632 18295 raft_consensus.cc:3037] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:37.024370 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:37.030521 18295 raft_consensus.cc:2454] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:37.032001 18369 tablet_bootstrap.cc:492] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:37.036579 18365 tablet_bootstrap.cc:492] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
W20260807 03:38:37.040927 18141 leader_election.cc:341] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:37.058867 18365 tablet_bootstrap.cc:654] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:37.063807 18141 leader_election.cc:302] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:37.034107 18950 raft_consensus.cc:504] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:37.097102 18991 consensus_queue.cc:1035] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.035s
I20260807 03:38:37.093240 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:37.110342 18884 raft_consensus.cc:2781] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:37.110909 18884 raft_consensus.cc:686] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:37.111958 18884 consensus_queue.cc:237] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:37.136538 18950 leader_election.cc:288] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
W20260807 03:38:37.164485 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 61185.4us
W20260807 03:38:37.170711 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 107362us
W20260807 03:38:37.190660 18027 leader_election.cc:341] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:37.200277 18366 tablet_bootstrap.cc:492] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:37.200918 18366 ts_tablet_manager.cc:1397] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.519s user 0.027s sys 0.027s
W20260807 03:38:37.208214 19001 log.cc:924] Time spent T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.214s user 0.004s sys 0.007s
I20260807 03:38:37.198895 18655 consensus_queue.cc:1035] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.020s
I20260807 03:38:37.220958 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:37.221159 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:37.226788 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:38:37.227411 18249 leader_election.cc:341] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:37.228578 18753 consensus_queue.cc:1035] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.023s
I20260807 03:38:37.224799 18760 consensus_queue.cc:1035] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.027s
W20260807 03:38:37.284691 18248 leader_election.cc:341] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:37.286562 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:37.285979 18248 leader_election.cc:302] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:37.303396 18623 raft_consensus.cc:2726] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260807 03:38:37.305648 19019 log.cc:924] Time spent T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.091s user 0.000s sys 0.001s
I20260807 03:38:37.302724 18366 raft_consensus.cc:348] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:37.313050 18369 tablet_bootstrap.cc:654] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:37.315464 18366 raft_consensus.cc:374] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:37.315862 18366 raft_consensus.cc:729] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:37.284952 18187 raft_consensus.cc:2454] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:37.323206 18366 consensus_queue.cc:260] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:37.327623 18140 leader_election.cc:341] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:37.334261 18897 raft_consensus.cc:482] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:37.362607 17914 catalog_manager.cc:5495] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 1 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
W20260807 03:38:37.416606 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 70449.6us
I20260807 03:38:37.417968 18366 ts_tablet_manager.cc:1428] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.217s user 0.009s sys 0.000s
I20260807 03:38:37.409659 18945 consensus_queue.cc:1035] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.088s
I20260807 03:38:37.361375 18897 raft_consensus.cc:504] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:37.489609 18081 raft_consensus.cc:3037] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:37.554656 18366 tablet_bootstrap.cc:492] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:37.679580 18991 raft_consensus.cc:482] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:37.797818 18991 raft_consensus.cc:504] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:37.800804 18365 tablet_bootstrap.cc:492] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:37.801331 18365 ts_tablet_manager.cc:1397] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.781s user 0.014s sys 0.031s
I20260807 03:38:37.855530 18897 leader_election.cc:288] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:37.855746 18991 leader_election.cc:288] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:37.905766 18365 raft_consensus.cc:348] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:37.906776 18365 raft_consensus.cc:374] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:37.907123 18365 raft_consensus.cc:729] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:37.910338 18365 consensus_queue.cc:260] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:37.927315 18003 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260807 03:38:37.931017 18189 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.503s user 0.002s sys 0.015s
I20260807 03:38:37.945533 18365 ts_tablet_manager.cc:1428] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.144s user 0.004s sys 0.005s
I20260807 03:38:37.957994 18771 raft_consensus.cc:482] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:37.958557 18771 raft_consensus.cc:504] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:37.959384 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:37.964613 18756 consensus_queue.cc:1035] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:37.967758 18365 tablet_bootstrap.cc:492] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:37.977468 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:37.985284 18771 leader_election.cc:288] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:38:37.986678 18026 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.576s user 0.000s sys 0.021s
W20260807 03:38:37.987010 18026 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.630s user 0.000s sys 0.021s
I20260807 03:38:37.999348 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:38.012418 19045 raft_consensus.cc:482] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:38.013078 19045 raft_consensus.cc:504] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:38.015215 19045 leader_election.cc:288] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:38.019861 18760 consensus_queue.cc:1035] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 03:38:38.019508 18027 leader_election.cc:302] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:38.024467 18973 raft_consensus.cc:2726] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:38.027623 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:38.028281 18191 raft_consensus.cc:2454] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 0.
I20260807 03:38:38.029320 18369 tablet_bootstrap.cc:492] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:38.029767 18369 ts_tablet_manager.cc:1397] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 1.011s user 0.023s sys 0.022s
I20260807 03:38:38.030802 18249 leader_election.cc:302] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
W20260807 03:38:38.033285 18141 leader_election.cc:341] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:38.040717 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:38.112978 19020 raft_consensus.cc:2781] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:38.113513 19020 raft_consensus.cc:482] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:38.113853 19020 raft_consensus.cc:3037] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
W20260807 03:38:38.129434 19062 log.cc:924] Time spent T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20260807 03:38:38.174664 18366 tablet_bootstrap.cc:700] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent opening tablet: real 0.133s user 0.006s sys 0.005s
I20260807 03:38:38.181222 18369 raft_consensus.cc:348] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:38.183503 18248 consensus_peers.cc:487] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 -> Peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Couldn't send request to peer d9716e2bf06442f5b3b0e81087c53c4d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260807 03:38:38.181612 18140 leader_election.cc:341] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
W20260807 03:38:38.182430 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 138407us
I20260807 03:38:38.185473 18140 leader_election.cc:302] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:38.188064 18365 tablet_bootstrap.cc:654] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:38.194350 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:38.183652 19047 raft_consensus.cc:482] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:38.183749 19004 consensus_queue.cc:1035] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.160s
I20260807 03:38:38.199363 18369 raft_consensus.cc:374] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:38.201720 18369 raft_consensus.cc:729] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:38.200572 19047 raft_consensus.cc:504] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:38.202375 18369 consensus_queue.cc:260] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:38.182513 18366 tablet_bootstrap.cc:654] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:38.205178 19047 leader_election.cc:288] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:38.206745 18369 ts_tablet_manager.cc:1428] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.177s user 0.006s sys 0.008s
I20260807 03:38:38.218133 19004 raft_consensus.cc:2726] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260807 03:38:38.230984 18140 consensus_peers.cc:487] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 -> Peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Couldn't send request to peer d9716e2bf06442f5b3b0e81087c53c4d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260807 03:38:38.191864 19020 raft_consensus.cc:504] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:38.241393 18369 tablet_bootstrap.cc:492] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:38.243417 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:38:38.264940 18140 leader_election.cc:341] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:38.265513 18140 leader_election.cc:302] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:38.251441 18964 raft_consensus.cc:482] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
W20260807 03:38:38.279498 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 87817.7us
I20260807 03:38:38.281659 18297 raft_consensus.cc:2379] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 1: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
W20260807 03:38:38.282452 19070 log.cc:924] Time spent T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.068s user 0.000s sys 0.001s
I20260807 03:38:38.282790 19041 raft_consensus.cc:2726] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:38.283892 18964 raft_consensus.cc:504] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:38.285868 19050 raft_consensus.cc:482] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:38.286404 18964 leader_election.cc:288] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:38:38.288507 18140 consensus_peers.cc:487] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 -> Peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Couldn't send request to peer d9716e2bf06442f5b3b0e81087c53c4d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260807 03:38:38.295171 18191 raft_consensus.cc:3037] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:38.291527 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:38.295217 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ca0223f47b46fd955704eba5259370" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:38.297430 19048 consensus_queue.cc:1035] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260807 03:38:38.296240 18956 raft_consensus.cc:482] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:38.299078 18956 raft_consensus.cc:504] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:38.319514 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ca0223f47b46fd955704eba5259370" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:38.326934 19020 leader_election.cc:288] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:38.286350 19050 raft_consensus.cc:504] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:38.341691 19050 leader_election.cc:288] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:38.342193 18956 leader_election.cc:288] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:38.344445 18027 leader_election.cc:302] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:38.392498 18365 tablet_bootstrap.cc:492] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
W20260807 03:38:38.392836 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 44164us
I20260807 03:38:38.352304 18964 raft_consensus.cc:2726] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:38.393020 18365 ts_tablet_manager.cc:1397] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.441s user 0.010s sys 0.025s
I20260807 03:38:38.401041 18369 tablet_bootstrap.cc:654] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:38.405387 19075 raft_consensus.cc:482] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:38.408874 19045 consensus_queue.cc:1035] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20260807 03:38:38.406494 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:38.417992 19075 raft_consensus.cc:504] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:38.406086 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:38.423815 19075 leader_election.cc:288] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:38.399384 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:38.452687 18189 raft_consensus.cc:3037] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:38.457767 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:38.466746 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:38.468701 18027 leader_election.cc:302] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
W20260807 03:38:38.469004 18140 leader_election.cc:341] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:38.469848 18140 leader_election.cc:302] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
W20260807 03:38:38.470584 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 79259us
I20260807 03:38:38.471736 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:38.472826 18897 raft_consensus.cc:2726] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:38.476918 18964 consensus_queue.cc:1035] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:38.440938 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:38:38.485167 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 90313.8us
W20260807 03:38:38.486613 18248 leader_election.cc:341] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:38.488034 18366 tablet_bootstrap.cc:492] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:38.488515 18366 ts_tablet_manager.cc:1397] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.987s user 0.018s sys 0.020s
I20260807 03:38:38.490736 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
W20260807 03:38:38.501482 18248 leader_election.cc:341] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:38.512306 19045 raft_consensus.cc:482] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:38.512776 19045 raft_consensus.cc:504] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:38.521467 18365 raft_consensus.cc:348] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:38.522348 18365 raft_consensus.cc:374] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:38.522670 18365 raft_consensus.cc:729] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:38.527977 18365 consensus_queue.cc:260] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:38.536271 18027 leader_election.cc:302] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:38.539837 18760 raft_consensus.cc:2726] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:38.543598 18366 raft_consensus.cc:348] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:38.545662 18366 raft_consensus.cc:374] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:38.546023 18366 raft_consensus.cc:729] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:38.546914 18366 consensus_queue.cc:260] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:38.553085 19045 leader_election.cc:288] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:38.557844 18366 ts_tablet_manager.cc:1428] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.069s user 0.007s sys 0.000s
I20260807 03:38:38.564216 18365 ts_tablet_manager.cc:1428] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.155s user 0.004s sys 0.004s
I20260807 03:38:38.575227 18366 tablet_bootstrap.cc:492] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:38.556457 18189 raft_consensus.cc:2454] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:38.619813 19052 raft_consensus.cc:2726] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:38.620424 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:38.628069 18365 tablet_bootstrap.cc:492] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
W20260807 03:38:38.646664 19094 log.cc:924] Time spent T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.084s user 0.000s sys 0.000s
I20260807 03:38:38.665040 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:38.671371 17912 catalog_manager.cc:5495] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 0 to 1, leader changed from <none> to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 1 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:38.751901 18973 raft_consensus.cc:482] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:38.752243 18369 tablet_bootstrap.cc:492] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:38.752693 18369 ts_tablet_manager.cc:1397] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.520s user 0.021s sys 0.005s
I20260807 03:38:38.752449 18973 raft_consensus.cc:504] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:38.755038 18973 leader_election.cc:288] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:38:38.799273 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 238981us
W20260807 03:38:38.810902 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 169721us
I20260807 03:38:38.809252 18960 raft_consensus.cc:482] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:38.816274 18960 raft_consensus.cc:504] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:38.818234 18960 leader_election.cc:288] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:38.818873 18366 tablet_bootstrap.cc:654] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:38.822978 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:38.831185 18249 leader_election.cc:302] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
W20260807 03:38:38.834643 18248 leader_election.cc:341] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:38.839211 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:38.842497 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:38.843152 19045 raft_consensus.cc:482] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:38.837675 18946 raft_consensus.cc:2781] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:38.843838 18946 raft_consensus.cc:686] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:38.843638 19045 raft_consensus.cc:504] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:38.844712 18946 consensus_queue.cc:237] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:38.848969 18141 leader_election.cc:302] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:38.854934 18027 leader_election.cc:302] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:38.856139 18973 raft_consensus.cc:2726] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:38.856809 18188 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44660 (request call id 245) took 1387 ms. Trace:
I20260807 03:38:38.857205 18188 rpcz_store.cc:276] 0807 03:38:37.469207 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:37.469397 (+ 190us) service_pool.cc:227] Handling call
0807 03:38:38.397655 (+928258us) spinlock_profiling.cc:243] Waited 3.26 ms on lock 0x7b5800003998. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f25f851 00007fe98f25f702 00007fe99814fb09 00007fe99708b730 00007fe99715179b 00007fe997151821 00007fe9971152b0 00007fe9971007f9 00007fe9971002a2 00007fe9907e6684
0807 03:38:38.397910 (+ 255us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:38:38.420494 (+ 22584us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:38:38.420548 (+ 54us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:38:38.526601 (+106053us) spinlock_profiling.cc:243] Waited 28.4 ms on lock 0x7b2c00047e58. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe990ba2e11 00007fe990bb2442 00007fe9921fac11 00007fe9921bf58a 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772
0807 03:38:38.544487 (+ 17886us) log.cc:842] Serialized 26 byte log entry
0807 03:38:38.771347 (+226860us) raft_consensus.cc:1631] Marking committed up to 0
0807 03:38:38.771400 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:38:38.771497 (+ 97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:38:38.856593 (+ 85096us) raft_consensus.cc:1671] finished
0807 03:38:38.856617 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:38:38.856735 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":11623,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":90,"spinlock_wait_cycles":72767744,"thread_start_us":226429,"threads_started":1,"wal-append.queue_time_us":283580}
I20260807 03:38:38.859907 18141 leader_election.cc:302] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:38.869956 19045 leader_election.cc:288] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:38.872227 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:38:38.873777 18248 leader_election.cc:341] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:38.876749 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:38.874676 18369 raft_consensus.cc:348] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:38.880985 18369 raft_consensus.cc:374] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:38.881366 18369 raft_consensus.cc:729] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:38.882133 18369 consensus_queue.cc:260] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:38.878800 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:38.883942 18954 raft_consensus.cc:2726] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:38.885649 18369 ts_tablet_manager.cc:1428] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.133s user 0.010s sys 0.004s
W20260807 03:38:38.893880 19109 log.cc:924] Time spent T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260807 03:38:38.900221 18369 tablet_bootstrap.cc:492] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:38.868626 18991 raft_consensus.cc:2726] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:38.916287 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:38.951754 18365 tablet_bootstrap.cc:654] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
W20260807 03:38:38.954734 18140 leader_election.cc:341] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:38.962276 18080 raft_consensus.cc:3037] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:38.959792 18293 raft_consensus.cc:2454] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 0.
I20260807 03:38:38.976598 18141 leader_election.cc:302] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:38.983680 19052 raft_consensus.cc:2781] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:39.009442 19052 raft_consensus.cc:482] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:39.009801 19052 raft_consensus.cc:3037] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:38.975589 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:38:39.025321 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 48810.3us
I20260807 03:38:39.034214 17914 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39494 (request call id 98) took 1833 ms. Trace:
I20260807 03:38:39.034813 17914 rpcz_store.cc:276] 0807 03:38:37.204643 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:37.205447 (+ 804us) service_pool.cc:227] Handling call
0807 03:38:39.025882 (+1820435us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 03:38:37.988137 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 03:38:37.988825 (+ 688us) write_op.cc:432] Acquiring schema lock in shared mode
0807 03:38:37.988873 (+ 48us) write_op.cc:435] Acquired schema lock
0807 03:38:37.988900 (+ 27us) tablet.cc:660] Decoding operations
0807 03:38:37.989390 (+ 490us) write_op.cc:620] Acquiring the partition lock for write op
0807 03:38:37.989463 (+ 73us) write_op.cc:641] Partition lock acquired for write op
0807 03:38:37.989495 (+ 32us) tablet.cc:683] Acquiring locks for 1 operations
0807 03:38:37.989704 (+ 209us) tablet.cc:699] Row locks acquired
0807 03:38:37.989720 (+ 16us) write_op.cc:260] PREPARE: finished
0807 03:38:37.989882 (+ 162us) write_op.cc:270] Start()
0807 03:38:37.989997 (+ 115us) write_op.cc:276] Timestamp: P: 1786073917989824 usec, L: 0
0807 03:38:37.990021 (+ 24us) op_driver.cc:331] REPLICATION: starting
0807 03:38:37.991123 (+ 1102us) log.cc:842] Serialized 543 byte log entry
0807 03:38:38.017635 (+ 26512us) op_driver.cc:444] REPLICATION: finished
0807 03:38:38.019157 (+ 1522us) write_op.cc:301] APPLY: starting
0807 03:38:38.019284 (+ 127us) tablet.cc:1365] starting BulkCheckPresence
0807 03:38:38.019413 (+ 129us) tablet.cc:1368] finished BulkCheckPresence
0807 03:38:38.019544 (+ 131us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 03:38:38.021374 (+ 1830us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 03:38:38.021523 (+ 149us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0807 03:38:38.021579 (+ 56us) write_op.cc:312] APPLY: finished
0807 03:38:38.021796 (+ 217us) log.cc:842] Serialized 22 byte log entry
0807 03:38:38.031676 (+ 9880us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:38:38.031754 (+ 78us) write_op.cc:454] Released schema lock
0807 03:38:38.031789 (+ 35us) write_op.cc:341] FINISH: Updating metrics
0807 03:38:38.036899 (+ 5110us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:38:38.039413 (+ 2514us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":4668,"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":1250,"apply.run_cpu_time_us":1988,"apply.run_wall_time_us":22951,"num_ops":1,"prepare.queue_time_us":515776,"prepare.run_cpu_time_us":2892,"prepare.run_wall_time_us":3279,"replication_time_us":27508,"thread_start_us":447694,"threads_started":2}]]}
I20260807 03:38:39.119406 18366 tablet_bootstrap.cc:492] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:39.119928 18366 ts_tablet_manager.cc:1397] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.555s user 0.018s sys 0.025s
I20260807 03:38:39.122844 18366 raft_consensus.cc:348] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:39.123577 18366 raft_consensus.cc:374] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:39.123871 18366 raft_consensus.cc:729] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:39.124684 18366 consensus_queue.cc:260] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:39.125021 18369 tablet_bootstrap.cc:654] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
W20260807 03:38:39.126880 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 72605.8us
I20260807 03:38:39.143003 18366 ts_tablet_manager.cc:1428] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20260807 03:38:39.141289 19052 raft_consensus.cc:504] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:39.145753 19052 leader_election.cc:288] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:38:39.160169 18249 leader_election.cc:341] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:39.160733 18249 leader_election.cc:302] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:39.162904 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:39.165607 18298 raft_consensus.cc:3037] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:39.163340 18753 raft_consensus.cc:2726] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:39.163568 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
W20260807 03:38:39.171684 18140 leader_election.cc:341] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:39.176337 18366 tablet_bootstrap.cc:492] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:39.193256 18365 tablet_bootstrap.cc:492] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:39.193729 18365 ts_tablet_manager.cc:1397] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.584s user 0.023s sys 0.013s
I20260807 03:38:39.196046 18365 raft_consensus.cc:348] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:39.201610 18365 raft_consensus.cc:374] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:39.201856 18365 raft_consensus.cc:729] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:39.203706 18365 consensus_queue.cc:260] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:39.205621 18249 leader_election.cc:341] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:39.206032 18249 leader_election.cc:302] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:39.206926 19104 raft_consensus.cc:2726] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:39.240756 18897 raft_consensus.cc:482] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:39.241870 18897 raft_consensus.cc:504] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:39.243891 18897 leader_election.cc:288] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:39.245040 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:39.245700 18369 tablet_bootstrap.cc:492] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:39.246132 18369 ts_tablet_manager.cc:1397] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.346s user 0.014s sys 0.005s
I20260807 03:38:39.248188 18366 tablet_bootstrap.cc:654] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
W20260807 03:38:39.250227 18027 leader_election.cc:341] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:39.252682 18369 raft_consensus.cc:348] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:39.253357 18369 raft_consensus.cc:374] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:39.253607 18369 raft_consensus.cc:729] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:39.256497 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:39.257094 18185 raft_consensus.cc:2454] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 0.
I20260807 03:38:39.259727 18365 ts_tablet_manager.cc:1428] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.066s user 0.011s sys 0.003s
I20260807 03:38:39.260684 18365 tablet_bootstrap.cc:492] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:39.263685 18369 consensus_queue.cc:260] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:39.266880 18369 ts_tablet_manager.cc:1428] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.021s user 0.007s sys 0.000s
I20260807 03:38:39.267802 18369 tablet_bootstrap.cc:492] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
I20260807 03:38:39.269033 17912 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39492 (request call id 106) took 1958 ms. Trace:
I20260807 03:38:39.270591 18298 raft_consensus.cc:2454] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:39.271768 18141 leader_election.cc:302] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:39.269407 17912 rpcz_store.cc:276] 0807 03:38:37.311014 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:37.311318 (+ 304us) service_pool.cc:227] Handling call
0807 03:38:39.268904 (+1957586us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 03:38:38.829967 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 03:38:38.839563 (+ 9596us) write_op.cc:432] Acquiring schema lock in shared mode
0807 03:38:38.839622 (+ 59us) write_op.cc:435] Acquired schema lock
0807 03:38:38.839646 (+ 24us) tablet.cc:660] Decoding operations
0807 03:38:38.840136 (+ 490us) write_op.cc:620] Acquiring the partition lock for write op
0807 03:38:38.840231 (+ 95us) write_op.cc:641] Partition lock acquired for write op
0807 03:38:38.840268 (+ 37us) tablet.cc:683] Acquiring locks for 1 operations
0807 03:38:38.840491 (+ 223us) tablet.cc:699] Row locks acquired
0807 03:38:38.840520 (+ 29us) write_op.cc:260] PREPARE: finished
0807 03:38:38.840689 (+ 169us) write_op.cc:270] Start()
0807 03:38:38.840825 (+ 136us) write_op.cc:276] Timestamp: P: 1786073918840640 usec, L: 0
0807 03:38:38.840852 (+ 27us) op_driver.cc:331] REPLICATION: starting
0807 03:38:38.841511 (+ 659us) log.cc:842] Serialized 543 byte log entry
0807 03:38:38.992077 (+150566us) op_driver.cc:444] REPLICATION: finished
0807 03:38:39.251188 (+259111us) write_op.cc:301] APPLY: starting
0807 03:38:39.252565 (+ 1377us) tablet.cc:1365] starting BulkCheckPresence
0807 03:38:39.258406 (+ 5841us) tablet.cc:1368] finished BulkCheckPresence
0807 03:38:39.259215 (+ 809us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 03:38:39.259408 (+ 193us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 03:38:39.259547 (+ 139us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0807 03:38:39.259587 (+ 40us) write_op.cc:312] APPLY: finished
0807 03:38:39.260305 (+ 718us) log.cc:842] Serialized 22 byte log entry
0807 03:38:39.260472 (+ 167us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:38:39.260549 (+ 77us) write_op.cc:454] Released schema lock
0807 03:38:39.260587 (+ 38us) write_op.cc:341] FINISH: Updating metrics
0807 03:38:39.260816 (+ 229us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:38:39.260855 (+ 39us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":344902,"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":188473,"apply.run_cpu_time_us":1994,"apply.run_wall_time_us":47567,"num_ops":1,"prepare.queue_time_us":135796,"prepare.run_cpu_time_us":3085,"prepare.run_wall_time_us":11800,"replication_time_us":146685,"thread_start_us":261932,"threads_started":2}]]}
I20260807 03:38:39.273519 18027 leader_election.cc:302] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:39.276572 19052 raft_consensus.cc:2781] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:39.276823 18655 raft_consensus.cc:2781] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:39.277066 19052 raft_consensus.cc:686] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:39.277290 18655 raft_consensus.cc:482] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:39.277727 18655 raft_consensus.cc:3037] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:39.276930 17920 catalog_manager.cc:5495] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 1 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:38:39.278020 19052 consensus_queue.cc:237] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:39.284015 18003 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 03:38:39.299059 17920 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39510 (request call id 106) took 2080 ms. Trace:
I20260807 03:38:39.300905 18366 tablet_bootstrap.cc:492] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:39.301438 18366 ts_tablet_manager.cc:1397] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.128s user 0.012s sys 0.019s
I20260807 03:38:39.302074 18655 raft_consensus.cc:504] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:39.300086 17920 rpcz_store.cc:276] 0807 03:38:37.218352 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:37.218655 (+ 303us) service_pool.cc:227] Handling call
0807 03:38:39.299000 (+2080345us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 03:38:39.283831 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 03:38:39.285870 (+ 2039us) write_op.cc:432] Acquiring schema lock in shared mode
0807 03:38:39.285909 (+ 39us) write_op.cc:435] Acquired schema lock
0807 03:38:39.285928 (+ 19us) tablet.cc:660] Decoding operations
0807 03:38:39.286308 (+ 380us) write_op.cc:620] Acquiring the partition lock for write op
0807 03:38:39.286375 (+ 67us) write_op.cc:641] Partition lock acquired for write op
0807 03:38:39.286421 (+ 46us) tablet.cc:683] Acquiring locks for 1 operations
0807 03:38:39.286627 (+ 206us) tablet.cc:699] Row locks acquired
0807 03:38:39.286647 (+ 20us) write_op.cc:260] PREPARE: finished
0807 03:38:39.286805 (+ 158us) write_op.cc:270] Start()
0807 03:38:39.286912 (+ 107us) write_op.cc:276] Timestamp: P: 1786073919286759 usec, L: 0
0807 03:38:39.286938 (+ 26us) op_driver.cc:331] REPLICATION: starting
0807 03:38:39.287592 (+ 654us) log.cc:842] Serialized 543 byte log entry
0807 03:38:39.293535 (+ 5943us) op_driver.cc:444] REPLICATION: finished
0807 03:38:39.293885 (+ 350us) write_op.cc:301] APPLY: starting
0807 03:38:39.293948 (+ 63us) tablet.cc:1365] starting BulkCheckPresence
0807 03:38:39.294046 (+ 98us) tablet.cc:1368] finished BulkCheckPresence
0807 03:38:39.294156 (+ 110us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 03:38:39.294350 (+ 194us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 03:38:39.294493 (+ 143us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0807 03:38:39.294531 (+ 38us) write_op.cc:312] APPLY: finished
0807 03:38:39.294693 (+ 162us) log.cc:842] Serialized 22 byte log entry
0807 03:38:39.294819 (+ 126us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:38:39.294907 (+ 88us) write_op.cc:454] Released schema lock
0807 03:38:39.294936 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0807 03:38:39.295160 (+ 224us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:38:39.295186 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":100301,"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":124,"apply.run_cpu_time_us":1481,"apply.run_wall_time_us":1482,"num_ops":1,"prepare.queue_time_us":109,"prepare.run_cpu_time_us":2722,"prepare.run_wall_time_us":6653,"replication_time_us":6509}]]}
I20260807 03:38:39.303886 18366 raft_consensus.cc:348] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:39.304564 18366 raft_consensus.cc:374] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:39.304801 18366 raft_consensus.cc:729] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:39.305408 18366 consensus_queue.cc:260] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:39.306494 18365 tablet_bootstrap.cc:654] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:39.307916 18366 ts_tablet_manager.cc:1428] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260807 03:38:39.308874 18366 tablet_bootstrap.cc:492] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:39.313519 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:39.314116 18185 raft_consensus.cc:3037] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:39.314435 18366 tablet_bootstrap.cc:654] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:39.327792 17912 catalog_manager.cc:5495] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 1 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
W20260807 03:38:39.348814 18248 consensus_peers.cc:487] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 -> Peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Couldn't send request to peer d9716e2bf06442f5b3b0e81087c53c4d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260807 03:38:39.350019 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:39.345532 18655 leader_election.cc:288] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:39.350724 18080 raft_consensus.cc:3037] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
W20260807 03:38:39.355001 18027 leader_election.cc:341] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:39.367667 18369 tablet_bootstrap.cc:654] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:39.372745 18366 tablet_bootstrap.cc:492] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:39.373191 18366 ts_tablet_manager.cc:1397] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.065s user 0.023s sys 0.011s
I20260807 03:38:39.400314 18366 raft_consensus.cc:348] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:39.401050 18366 raft_consensus.cc:374] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:39.401309 18366 raft_consensus.cc:729] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:39.401854 18366 consensus_queue.cc:260] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:39.419238 18185 raft_consensus.cc:2454] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:38:39.433223 19104 raft_consensus.cc:482] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:39.437253 18027 leader_election.cc:302] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:39.437157 18991 raft_consensus.cc:482] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:39.441385 18991 raft_consensus.cc:504] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:39.443751 18991 leader_election.cc:288] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:39.451900 18366 ts_tablet_manager.cc:1428] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.078s user 0.006s sys 0.001s
I20260807 03:38:39.434922 19104 raft_consensus.cc:504] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:39.452419 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:39.606034 18365 tablet_bootstrap.cc:492] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:39.624727 18365 ts_tablet_manager.cc:1397] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.364s user 0.031s sys 0.050s
I20260807 03:38:39.448004 19048 consensus_queue.cc:1035] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
W20260807 03:38:39.617322 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 180142us
I20260807 03:38:39.627321 18300 raft_consensus.cc:2454] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 0.
I20260807 03:38:39.618023 19056 raft_consensus.cc:2781] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:39.631546 19104 leader_election.cc:288] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:39.632112 18366 tablet_bootstrap.cc:492] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:39.632440 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:39.635851 19056 raft_consensus.cc:686] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:38:39.637022 19056 consensus_queue.cc:237] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:39.638476 18248 leader_election.cc:341] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:39.618333 18369 tablet_bootstrap.cc:492] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:39.645877 18369 ts_tablet_manager.cc:1397] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.378s user 0.016s sys 0.021s
I20260807 03:38:39.656718 18369 raft_consensus.cc:348] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:39.657557 18369 raft_consensus.cc:374] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:39.657828 18369 raft_consensus.cc:729] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:39.658522 18369 consensus_queue.cc:260] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:39.661109 18897 raft_consensus.cc:482] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:39.661656 18897 raft_consensus.cc:504] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:39.663113 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:39.664113 18897 leader_election.cc:288] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:39.686290 18366 tablet_bootstrap.cc:654] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:39.697721 18365 raft_consensus.cc:348] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:39.698467 18365 raft_consensus.cc:374] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:39.698704 18365 raft_consensus.cc:729] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:39.699286 18365 consensus_queue.cc:260] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:39.701989 18365 ts_tablet_manager.cc:1428] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.077s user 0.003s sys 0.002s
I20260807 03:38:39.703186 18365 tablet_bootstrap.cc:492] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02: Bootstrap starting.
I20260807 03:38:39.707595 18369 ts_tablet_manager.cc:1428] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.061s user 0.006s sys 0.002s
I20260807 03:38:39.712952 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:39.713670 18188 raft_consensus.cc:2454] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 0.
I20260807 03:38:39.717178 17912 catalog_manager.cc:5495] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 1 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:39.732348 18141 leader_election.cc:302] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:39.740726 18249 leader_election.cc:302] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:39.742504 18369 tablet_bootstrap.cc:492] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49: Bootstrap starting.
W20260807 03:38:39.743510 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 109322us
I20260807 03:38:39.745085 19090 raft_consensus.cc:2781] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:39.745465 19090 raft_consensus.cc:482] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:39.747565 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:39.748435 18309 raft_consensus.cc:2379] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 1: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:38:39.746929 19090 raft_consensus.cc:3037] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:39.761452 18365 tablet_bootstrap.cc:654] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02: Neither blocks nor log segments found. Creating new log.
W20260807 03:38:39.771080 18140 leader_election.cc:341] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:39.744032 19150 raft_consensus.cc:2781] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:39.779968 19150 raft_consensus.cc:482] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:39.797309 18079 raft_consensus.cc:3037] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:39.800254 19150 raft_consensus.cc:3037] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:39.794397 17914 catalog_manager.cc:5495] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 1 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:38:39.796691 19086 raft_consensus.cc:482] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:39.818985 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:39.822403 18027 leader_election.cc:302] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:39.828320 19090 raft_consensus.cc:504] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:39.824862 19086 raft_consensus.cc:504] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:39.844070 18369 tablet_bootstrap.cc:654] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:39.818226 19149 consensus_queue.cc:1035] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.029s
I20260807 03:38:39.852835 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:39.853477 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:39.846037 19086 leader_election.cc:288] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:39.869427 18365 tablet_bootstrap.cc:492] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02: No bootstrap required, opened a new log
I20260807 03:38:39.870146 18365 ts_tablet_manager.cc:1397] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02: Time spent bootstrapping tablet: real 0.167s user 0.013s sys 0.008s
I20260807 03:38:39.836472 19090 leader_election.cc:288] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:39.870218 17912 catalog_manager.cc:5495] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 0 to 1, leader changed from <none> to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 1 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:39.854490 19143 raft_consensus.cc:2726] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260807 03:38:39.882136 18248 leader_election.cc:341] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
W20260807 03:38:39.896561 18248 leader_election.cc:341] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
W20260807 03:38:39.897109 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 66781.5us
I20260807 03:38:39.905337 19099 consensus_queue.cc:1035] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:39.907971 19150 raft_consensus.cc:504] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:39.929260 19150 leader_election.cc:288] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:39.930703 18365 raft_consensus.cc:348] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:39.931536 18365 raft_consensus.cc:374] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:39.931815 18365 raft_consensus.cc:729] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Initialized, Role: FOLLOWER
I20260807 03:38:39.932444 18365 consensus_queue.cc:260] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:39.934619 18366 tablet_bootstrap.cc:492] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:39.935078 18366 ts_tablet_manager.cc:1397] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.303s user 0.022s sys 0.005s
I20260807 03:38:39.937497 18365 ts_tablet_manager.cc:1428] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02: Time spent starting tablet: real 0.067s user 0.007s sys 0.000s
I20260807 03:38:39.940716 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:39.941085 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:39.942651 18187 raft_consensus.cc:2379] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 1: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:38:39.943081 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:39.943742 18180 raft_consensus.cc:2454] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 0.
W20260807 03:38:39.950204 18140 leader_election.cc:341] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:39.949110 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:39.960510 18249 leader_election.cc:302] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:39.962711 18295 raft_consensus.cc:2379] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 1: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:38:39.969820 18141 leader_election.cc:302] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:39.972990 18249 leader_election.cc:302] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:39.960552 18366 raft_consensus.cc:348] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:39.974599 18366 raft_consensus.cc:374] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:39.974876 18366 raft_consensus.cc:729] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:39.975589 18366 consensus_queue.cc:260] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:39.975903 19163 raft_consensus.cc:2726] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260807 03:38:39.984586 18369 tablet_bootstrap.cc:492] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49: No bootstrap required, opened a new log
I20260807 03:38:39.985052 18369 ts_tablet_manager.cc:1397] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49: Time spent bootstrapping tablet: real 0.246s user 0.009s sys 0.024s
I20260807 03:38:39.971910 19090 raft_consensus.cc:2726] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260807 03:38:39.978729 19007 raft_consensus.cc:2781] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:39.988924 19007 raft_consensus.cc:482] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:39.989252 19007 raft_consensus.cc:3037] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:39.991065 19163 raft_consensus.cc:482] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:39.992594 19163 raft_consensus.cc:504] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:39.995141 19163 leader_election.cc:288] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:40.000916 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:40.008772 19007 raft_consensus.cc:504] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:40.006428 18140 leader_election.cc:341] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:40.011317 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:40.011912 18069 raft_consensus.cc:3037] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:40.017335 18369 raft_consensus.cc:348] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:40.018337 18369 raft_consensus.cc:374] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:40.018658 18369 raft_consensus.cc:729] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Initialized, Role: FOLLOWER
I20260807 03:38:40.023147 18069 raft_consensus.cc:2454] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:40.019621 19007 leader_election.cc:288] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:40.024338 18248 leader_election.cc:302] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:40.025146 19007 raft_consensus.cc:2781] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:40.027629 19007 raft_consensus.cc:686] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:40.029486 19007 consensus_queue.cc:237] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:40.025733 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:40.031289 18300 raft_consensus.cc:2454] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 0.
I20260807 03:38:40.034034 18366 ts_tablet_manager.cc:1428] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.099s user 0.004s sys 0.004s
I20260807 03:38:40.033890 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:40.034813 18187 raft_consensus.cc:3037] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:40.035324 18366 tablet_bootstrap.cc:492] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:40.042250 18141 leader_election.cc:302] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:40.047778 19163 raft_consensus.cc:2781] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:40.048175 19163 raft_consensus.cc:482] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:40.048493 19163 raft_consensus.cc:3037] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:40.052176 17914 catalog_manager.cc:5495] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 1 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:38:40.054731 19163 raft_consensus.cc:504] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:40.056704 19163 leader_election.cc:288] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:40.048502 18187 raft_consensus.cc:2454] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:40.058863 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:40.059437 18300 raft_consensus.cc:3037] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:40.062187 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
W20260807 03:38:40.064131 18140 leader_election.cc:341] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:40.019394 18369 consensus_queue.cc:260] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:40.075309 18369 ts_tablet_manager.cc:1428] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49: Time spent starting tablet: real 0.090s user 0.007s sys 0.000s
I20260807 03:38:40.077857 18366 tablet_bootstrap.cc:654] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:40.075152 18300 raft_consensus.cc:2454] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:40.081876 18141 leader_election.cc:302] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:40.082859 18881 raft_consensus.cc:2781] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:40.083315 18881 raft_consensus.cc:686] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:40.084026 18881 consensus_queue.cc:237] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:40.103820 18366 tablet_bootstrap.cc:492] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:40.104300 18366 ts_tablet_manager.cc:1397] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.069s user 0.011s sys 0.005s
I20260807 03:38:40.107385 18366 raft_consensus.cc:348] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:40.108032 18366 raft_consensus.cc:374] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:40.108300 18366 raft_consensus.cc:729] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:40.108867 18366 consensus_queue.cc:260] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:40.114064 17912 catalog_manager.cc:5495] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 1 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:40.121443 18366 ts_tablet_manager.cc:1428] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20260807 03:38:40.122366 18366 tablet_bootstrap.cc:492] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:40.127731 18366 tablet_bootstrap.cc:654] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:40.142007 18366 tablet_bootstrap.cc:492] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:40.142445 18366 ts_tablet_manager.cc:1397] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.004s
I20260807 03:38:40.144831 18366 raft_consensus.cc:348] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:40.145448 18366 raft_consensus.cc:374] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:40.145721 18366 raft_consensus.cc:729] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:40.146308 18366 consensus_queue.cc:260] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:40.148193 18366 ts_tablet_manager.cc:1428] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260807 03:38:40.149601 18366 tablet_bootstrap.cc:492] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:40.154930 18366 tablet_bootstrap.cc:654] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:40.161046 19105 raft_consensus.cc:482] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:40.161657 19105 raft_consensus.cc:504] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:40.165699 18366 tablet_bootstrap.cc:492] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:40.166121 18366 ts_tablet_manager.cc:1397] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.008s
I20260807 03:38:40.171813 18366 raft_consensus.cc:348] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:40.172700 18366 raft_consensus.cc:374] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:40.172961 18366 raft_consensus.cc:729] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:40.173544 18366 consensus_queue.cc:260] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:40.175921 18366 ts_tablet_manager.cc:1428] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20260807 03:38:40.176847 18366 tablet_bootstrap.cc:492] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:40.178397 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:40.182336 18366 tablet_bootstrap.cc:654] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
W20260807 03:38:40.184231 18140 leader_election.cc:341] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:40.163980 19105 leader_election.cc:288] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:40.193212 18366 tablet_bootstrap.cc:492] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:40.203457 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:40.204023 18366 ts_tablet_manager.cc:1397] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.027s user 0.012s sys 0.004s
I20260807 03:38:40.204329 18300 raft_consensus.cc:2454] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 0.
I20260807 03:38:40.206429 18366 raft_consensus.cc:348] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:40.207115 18366 raft_consensus.cc:374] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:40.207358 18366 raft_consensus.cc:729] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:40.211483 18141 leader_election.cc:302] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:40.213470 19105 raft_consensus.cc:2781] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:38:40.213877 19105 raft_consensus.cc:482] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:40.215129 19105 raft_consensus.cc:3037] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:40.215865 18295 raft_consensus.cc:3037] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:40.220396 19143 consensus_queue.cc:1035] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:40.224637 18366 consensus_queue.cc:260] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:40.226480 18366 ts_tablet_manager.cc:1428] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.022s user 0.005s sys 0.000s
I20260807 03:38:40.227420 18366 tablet_bootstrap.cc:492] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d: Bootstrap starting.
I20260807 03:38:40.246960 19143 consensus_queue.cc:1035] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:40.260833 18366 tablet_bootstrap.cc:654] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d: Neither blocks nor log segments found. Creating new log.
I20260807 03:38:40.279039 19105 raft_consensus.cc:504] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:40.282518 19105 leader_election.cc:288] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:40.283519 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:40.283700 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:40.284343 18309 raft_consensus.cc:3037] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 0 FOLLOWER]: Advancing to term 1
W20260807 03:38:40.290570 18140 leader_election.cc:341] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Illegal state: must be running to vote when last-logged opid is not known
I20260807 03:38:40.297144 18309 raft_consensus.cc:2454] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:40.306905 18141 leader_election.cc:302] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:40.314388 18366 tablet_bootstrap.cc:492] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d: No bootstrap required, opened a new log
I20260807 03:38:40.314851 18366 ts_tablet_manager.cc:1397] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent bootstrapping tablet: real 0.088s user 0.011s sys 0.002s
I20260807 03:38:40.318248 18366 raft_consensus.cc:348] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:40.320892 18366 raft_consensus.cc:374] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:38:40.325491 18366 raft_consensus.cc:729] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Initialized, Role: FOLLOWER
I20260807 03:38:40.326309 18366 consensus_queue.cc:260] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:40.330828 18366 ts_tablet_manager.cc:1428] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent starting tablet: real 0.016s user 0.005s sys 0.003s
I20260807 03:38:40.315452 19105 raft_consensus.cc:2781] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:38:40.336683 19105 raft_consensus.cc:686] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:40.337546 19105 consensus_queue.cc:237] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:40.343813 18003 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 03:38:40.357548 19056 raft_consensus.cc:482] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:40.358053 19056 raft_consensus.cc:504] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:40.360052 19056 leader_election.cc:288] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:40.361665 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:40.362447 18309 raft_consensus.cc:2379] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 1: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:38:40.362283 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:40.364040 18027 leader_election.cc:302] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:40.365236 19056 raft_consensus.cc:3037] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:40.387683 19056 raft_consensus.cc:2726] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:40.399936 18079 raft_consensus.cc:3037] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:40.411155 17920 catalog_manager.cc:5495] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 1 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:40.430800 19020 consensus_queue.cc:1035] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:40.454100 19077 consensus_queue.cc:1035] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:40.512933 19105 consensus_queue.cc:1035] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:40.536291 19105 consensus_queue.cc:1035] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:40.564270 19056 raft_consensus.cc:482] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:40.564800 19056 raft_consensus.cc:504] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:40.566558 19056 leader_election.cc:288] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:40.567660 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:40.567778 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:40.569733 18027 leader_election.cc:302] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:40.570577 19056 raft_consensus.cc:2726] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:40.571229 19056 raft_consensus.cc:482] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:40.571664 19056 raft_consensus.cc:504] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:40.574630 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:40.575389 18185 raft_consensus.cc:2379] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 1: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:38:40.577339 19056 leader_election.cc:288] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:40.578615 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:40.579392 18309 raft_consensus.cc:2379] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 1: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:38:40.580417 18027 leader_election.cc:302] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:40.581148 19056 raft_consensus.cc:3037] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:40.586030 19056 raft_consensus.cc:2726] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:40.588529 18075 raft_consensus.cc:3037] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:40.664484 18655 raft_consensus.cc:482] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:40.664973 18655 raft_consensus.cc:504] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:40.666606 18655 leader_election.cc:288] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:40.669042 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:40.669828 18309 raft_consensus.cc:2379] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 1: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:38:40.671514 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:40.673638 18027 leader_election.cc:302] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:40.674407 19056 raft_consensus.cc:3037] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:38:40.680472 19056 raft_consensus.cc:2726] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:38:40.768425 19163 consensus_queue.cc:1035] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:40.799340 19105 consensus_queue.cc:1035] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260807 03:38:41.380136 18003 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 03:38:41.943377 19105 raft_consensus.cc:482] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:41.943962 19105 raft_consensus.cc:504] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:41.946120 19105 leader_election.cc:288] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:41.946704 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:41.947009 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:41.947305 18070 raft_consensus.cc:2454] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:41.947700 18309 raft_consensus.cc:2454] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:41.948313 18140 leader_election.cc:302] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:41.949009 19105 raft_consensus.cc:2781] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:41.949322 19105 raft_consensus.cc:482] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:41.949581 19105 raft_consensus.cc:3037] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:41.955266 19105 raft_consensus.cc:504] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:41.957049 19105 leader_election.cc:288] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:41.958081 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:41.958529 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:41.958667 18309 raft_consensus.cc:3037] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:41.959064 18075 raft_consensus.cc:3037] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:41.963413 18075 raft_consensus.cc:2454] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:38:41.964195 18140 leader_election.cc:302] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:41.964885 19105 raft_consensus.cc:2781] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:38:41.965984 19105 raft_consensus.cc:686] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 2 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:41.967545 19105 consensus_queue.cc:237] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:41.977155 18309 raft_consensus.cc:2454] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:38:41.978132 17912 catalog_manager.cc:5495] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 0 to 2, leader changed from <none> to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 2 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:42.421423 18309 raft_consensus.cc:1261] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260807 03:38:42.422717 19105 consensus_queue.cc:1035] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:42.465545 18070 raft_consensus.cc:1261] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260807 03:38:42.467761 19163 consensus_queue.cc:1035] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:38:42.689654 16154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:38:42.722898 19219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:38:42.723171 19218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:38:42.724828 19223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:38:42.727491 16154 server_base.cc:1034] running on GCE node
I20260807 03:38:42.728986 16154 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260807 03:38:42.729305 16154 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260807 03:38:42.729509 16154 hybrid_clock.cc:648] HybridClock initialized: now 1786073922729493 us; error 0 us; skew 500 ppm
I20260807 03:38:42.730203 16154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:38:42.737470 16154 webserver.cc:460] Webserver started at http://127.15.198.132:33829/ using document root <none> and password file <none>
I20260807 03:38:42.737927 16154 fs_manager.cc:359] Metadata directory not provided
I20260807 03:38:42.738090 16154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:38:42.738334 16154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:38:42.739418 16154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786073845599630-16154-0/minicluster-data/ts-3-root/instance:
uuid: "5c9b5062bdfa46af9396f1a85025a331"
format_stamp: "Formatted at 2026-08-07 03:38:42 on dist-test-slave-7vjv"
I20260807 03:38:42.747051 16154 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260807 03:38:42.752725 19235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:38:42.753713 16154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 03:38:42.754012 16154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786073845599630-16154-0/minicluster-data/ts-3-root
uuid: "5c9b5062bdfa46af9396f1a85025a331"
format_stamp: "Formatted at 2026-08-07 03:38:42 on dist-test-slave-7vjv"
I20260807 03:38:42.754314 16154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786073845599630-16154-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786073845599630-16154-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786073845599630-16154-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:38:42.782061 16154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:38:42.783979 16154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:38:42.814590 16154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:38:42.818285 16154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:38:42.818493 16154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:38:42.818799 16154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:38:42.818951 16154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:38:42.939088 16154 rpc_server.cc:307] RPC server started. Bound to: 127.15.198.132:41381
I20260807 03:38:42.939194 19330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.198.132:41381 every 8 connection(s)
I20260807 03:38:42.964847 16154 auto_leader_rebalancer-test.cc:413] add a tserver: 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:38:42.978006 19331 heartbeater.cc:347] Connected to a master server at 127.15.198.190:40799
I20260807 03:38:42.978580 19331 heartbeater.cc:464] Registering TS with master...
I20260807 03:38:42.979553 19331 heartbeater.cc:511] Master 127.15.198.190:40799 requested a full tablet report, sending...
I20260807 03:38:42.983812 17912 ts_manager.cc:194] Registered new tserver with Master: 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:38:42.985742 17912 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35098
I20260807 03:38:44.212960 19349 raft_consensus.cc:482] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:44.216696 19349 raft_consensus.cc:504] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.195997 19142 raft_consensus.cc:482] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:44.220647 19352 raft_consensus.cc:482] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:44.238099 19142 raft_consensus.cc:504] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.242818 19142 leader_election.cc:288] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.224325 19354 raft_consensus.cc:482] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:44.268748 19243 raft_consensus.cc:482] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:44.253293 19352 raft_consensus.cc:504] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.290150 19353 raft_consensus.cc:482] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:44.290889 19353 raft_consensus.cc:504] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.292981 19338 raft_consensus.cc:482] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:44.293604 19338 raft_consensus.cc:504] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.279377 19243 raft_consensus.cc:504] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.303735 19142 raft_consensus.cc:482] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:44.304425 19142 raft_consensus.cc:504] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.262826 19354 raft_consensus.cc:504] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.322574 19347 raft_consensus.cc:482] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:44.323227 19347 raft_consensus.cc:504] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.331591 19347 leader_election.cc:288] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:44.332386 19338 leader_election.cc:288] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.334527 19353 leader_election.cc:288] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.335781 19142 leader_election.cc:288] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.335917 19344 raft_consensus.cc:482] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:44.336428 19344 raft_consensus.cc:504] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.331591 19243 leader_election.cc:288] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.338580 19344 leader_election.cc:288] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.340158 19354 leader_election.cc:288] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.342764 19344 raft_consensus.cc:482] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:44.357733 19243 raft_consensus.cc:482] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:44.358273 19243 raft_consensus.cc:504] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.359747 19344 raft_consensus.cc:504] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.343308 19347 raft_consensus.cc:482] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:44.335695 19349 leader_election.cc:288] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.343197 19357 raft_consensus.cc:482] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:44.364296 19347 raft_consensus.cc:504] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.362214 19142 raft_consensus.cc:482] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:44.390946 19056 raft_consensus.cc:482] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:44.391889 19056 raft_consensus.cc:504] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.377925 19331 heartbeater.cc:503] Master 127.15.198.190:40799 was elected leader, sending a full tablet report...
I20260807 03:38:44.366465 19243 leader_election.cc:288] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.397028 19335 raft_consensus.cc:482] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:44.396617 19243 raft_consensus.cc:482] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:44.391017 19354 raft_consensus.cc:482] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:44.397523 19335 raft_consensus.cc:504] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.397737 19354 raft_consensus.cc:504] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.399456 19335 leader_election.cc:288] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.399525 19056 leader_election.cc:288] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.399806 19354 leader_election.cc:288] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.397670 19243 raft_consensus.cc:504] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.368602 19142 raft_consensus.cc:504] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.402021 19056 raft_consensus.cc:482] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:44.406023 19056 raft_consensus.cc:504] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.351544 19352 leader_election.cc:288] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.409336 19056 leader_election.cc:288] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.409763 19056 raft_consensus.cc:482] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:44.410228 19056 raft_consensus.cc:504] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.364487 19340 raft_consensus.cc:482] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:44.367134 19347 leader_election.cc:288] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:44.368835 19361 raft_consensus.cc:482] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:44.415494 19056 leader_election.cc:288] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.418902 19056 raft_consensus.cc:482] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:44.419476 19056 raft_consensus.cc:504] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.405138 19354 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:44.423609 19341 raft_consensus.cc:482] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:44.401810 19364 raft_consensus.cc:482] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:44.414387 19352 raft_consensus.cc:482] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:44.415956 19369 raft_consensus.cc:482] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:44.436182 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.434608 19142 leader_election.cc:288] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.434417 19341 raft_consensus.cc:504] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.436779 18311 raft_consensus.cc:2454] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:38:44.443063 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.431731 19354 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.446393 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.446915 19335 raft_consensus.cc:482] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:44.450088 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.416491 19374 raft_consensus.cc:482] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:44.455035 19354 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.443434 19352 raft_consensus.cc:504] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.455224 19374 raft_consensus.cc:504] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.435966 19364 raft_consensus.cc:504] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.425598 19347 raft_consensus.cc:482] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:44.454609 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.452251 19056 leader_election.cc:288] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.462915 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.435951 19369 raft_consensus.cc:504] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.466456 19056 raft_consensus.cc:482] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:44.467494 19056 raft_consensus.cc:504] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.454648 19353 raft_consensus.cc:482] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:44.424379 19361 raft_consensus.cc:504] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.447431 19335 raft_consensus.cc:504] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.461185 19347 raft_consensus.cc:504] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.365283 19357 raft_consensus.cc:504] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.477481 19056 leader_election.cc:288] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.464880 19365 raft_consensus.cc:482] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:44.416055 19340 raft_consensus.cc:504] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.482101 19347 leader_election.cc:288] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:44.485574 19365 raft_consensus.cc:504] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.486189 19357 leader_election.cc:288] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.476028 19353 raft_consensus.cc:504] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.487668 19365 leader_election.cc:288] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.488272 19353 leader_election.cc:288] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.489002 19056 raft_consensus.cc:482] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:44.489410 19364 leader_election.cc:288] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.489456 19056 raft_consensus.cc:504] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.489962 19364 raft_consensus.cc:482] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:44.490376 19364 raft_consensus.cc:504] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.491102 19056 leader_election.cc:288] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.491277 19378 raft_consensus.cc:482] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:44.477890 19341 leader_election.cc:288] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.494642 18309 raft_consensus.cc:2454] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:38:44.493801 19378 raft_consensus.cc:504] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.496765 19378 leader_election.cc:288] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.477780 19374 leader_election.cc:288] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:44.497836 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.498502 19341 raft_consensus.cc:482] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:44.485003 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.498939 19341 raft_consensus.cc:504] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.500525 19369 leader_election.cc:288] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.500847 19341 leader_election.cc:288] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.484136 19344 leader_election.cc:288] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.511183 19377 raft_consensus.cc:482] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:44.511260 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.511792 19377 raft_consensus.cc:504] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.513413 19344 raft_consensus.cc:482] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:44.513897 19344 raft_consensus.cc:504] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.514477 19377 leader_election.cc:288] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:44.505790 19371 raft_consensus.cc:482] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:44.515561 19344 leader_election.cc:288] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.506448 19352 leader_election.cc:288] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.509151 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.515889 19371 raft_consensus.cc:504] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.506644 19243 leader_election.cc:288] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.527006 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.528343 19341 raft_consensus.cc:482] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:44.520955 19361 leader_election.cc:288] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:44.534511 18187 raft_consensus.cc:2454] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:44.522805 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.523083 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.549026 18300 raft_consensus.cc:2454] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:38:44.517999 19364 leader_election.cc:288] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.522215 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.521667 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.564515 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.564953 19371 leader_election.cc:288] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:44.521006 19335 leader_election.cc:288] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.603145 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.603916 18300 raft_consensus.cc:2454] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:38:44.606510 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.603065 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.608331 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.616716 19369 raft_consensus.cc:482] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:44.617095 18297 raft_consensus.cc:2454] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:38:44.619978 19340 leader_election.cc:288] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:44.620432 19368 raft_consensus.cc:482] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:44.603276 18293 raft_consensus.cc:2454] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 2.
I20260807 03:38:44.617228 19369 raft_consensus.cc:504] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.621111 19368 raft_consensus.cc:504] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.564978 19341 raft_consensus.cc:504] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.622526 19353 raft_consensus.cc:482] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:44.625923 19353 raft_consensus.cc:504] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.627170 19372 raft_consensus.cc:482] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:44.625273 19368 leader_election.cc:288] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.627666 19372 raft_consensus.cc:504] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.631855 19372 leader_election.cc:288] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.633472 19341 leader_election.cc:288] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.643741 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.644371 18187 raft_consensus.cc:2454] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:44.644392 19374 raft_consensus.cc:482] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:44.644857 19353 leader_election.cc:288] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.639462 19375 raft_consensus.cc:482] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:44.644943 19374 raft_consensus.cc:504] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.645512 19375 raft_consensus.cc:504] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.648053 19374 leader_election.cc:288] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:44.649011 19369 leader_election.cc:288] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.649340 19375 leader_election.cc:288] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:44.650629 19377 raft_consensus.cc:482] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:44.650930 19371 raft_consensus.cc:482] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:44.651244 19377 raft_consensus.cc:504] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.651418 19371 raft_consensus.cc:504] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.653641 19371 leader_election.cc:288] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:44.655526 19392 raft_consensus.cc:482] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:44.655995 19353 raft_consensus.cc:482] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:44.658051 19353 raft_consensus.cc:504] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.658628 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.658869 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.660526 19353 leader_election.cc:288] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.656072 19392 raft_consensus.cc:504] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.655470 19341 raft_consensus.cc:482] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:44.665743 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.661562 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.666400 18249 leader_election.cc:302] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:44.666286 19341 raft_consensus.cc:504] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.667263 19392 leader_election.cc:288] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.656631 19377 leader_election.cc:288] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:44.667289 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.657311 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.667474 19353 raft_consensus.cc:482] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:44.668213 18293 raft_consensus.cc:2454] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:38:44.668278 19353 raft_consensus.cc:504] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.669476 19056 raft_consensus.cc:482] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:44.669907 19056 raft_consensus.cc:504] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.671437 19056 leader_election.cc:288] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.671836 19056 raft_consensus.cc:482] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:44.671957 19364 raft_consensus.cc:482] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:44.673270 19364 raft_consensus.cc:504] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.674000 19056 raft_consensus.cc:504] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.674984 19364 leader_election.cc:288] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.675196 19353 leader_election.cc:288] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.675622 19056 leader_election.cc:288] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.672235 19243 raft_consensus.cc:482] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:44.676445 19390 raft_consensus.cc:2781] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:44.676826 19390 raft_consensus.cc:482] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:44.676633 19243 raft_consensus.cc:504] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.677206 19390 raft_consensus.cc:3037] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:44.678445 19243 leader_election.cc:288] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.678794 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:44.680337 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:44.681629 19347 raft_consensus.cc:482] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:44.681986 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:44.682123 19347 raft_consensus.cc:504] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.682538 18070 raft_consensus.cc:2454] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:44.683022 19341 leader_election.cc:288] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.684003 19347 leader_election.cc:288] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:44.687029 19341 raft_consensus.cc:482] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:44.687588 19344 raft_consensus.cc:482] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:44.687546 19341 raft_consensus.cc:504] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.688153 19344 raft_consensus.cc:504] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.689817 19341 leader_election.cc:288] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.689922 19344 leader_election.cc:288] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.689975 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:44.690402 19352 raft_consensus.cc:482] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:44.691051 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:44.690984 19352 raft_consensus.cc:504] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.692675 19347 raft_consensus.cc:482] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:44.693100 19352 leader_election.cc:288] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.695225 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:44.695559 19243 raft_consensus.cc:482] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:44.695818 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:44.696122 19243 raft_consensus.cc:504] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.696424 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:44.696557 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:44.694607 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:44.697954 19243 leader_election.cc:288] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.698510 19344 raft_consensus.cc:482] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:44.698872 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.698945 19344 raft_consensus.cc:504] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.699553 18297 raft_consensus.cc:2454] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:38:44.698362 19243 raft_consensus.cc:482] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:44.700410 19243 raft_consensus.cc:504] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.700891 19344 leader_election.cc:288] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.702034 19243 leader_election.cc:288] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.702713 19390 raft_consensus.cc:504] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.693143 19347 raft_consensus.cc:504] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.704799 19390 leader_election.cc:288] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:44.704988 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.705514 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:44.705782 19347 leader_election.cc:288] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:44.706733 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.706795 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.707068 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.707701 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.706123 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.708868 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.709090 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.709662 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.710399 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.711280 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.711509 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.711833 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.713073 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.718109 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.718603 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.719261 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.719698 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.710999 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.712435 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.716854 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.721195 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.717111 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.721390 18183 raft_consensus.cc:2454] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:38:44.722479 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.722548 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.722610 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.712630 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.724731 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:44.725291 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:44.726003 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:44.726402 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:44.711552 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.712558 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.720240 19352 raft_consensus.cc:482] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:44.727836 19384 raft_consensus.cc:482] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:44.727586 19352 raft_consensus.cc:504] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.727056 18249 leader_election.cc:302] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:44.728318 19384 raft_consensus.cc:504] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.729136 19347 raft_consensus.cc:2781] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:44.729471 19352 leader_election.cc:288] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.730118 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.730486 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:44.730906 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:44.731117 18079 raft_consensus.cc:2454] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:44.731400 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:44.731938 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:44.731967 18075 raft_consensus.cc:2454] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:44.734797 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:44.736058 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:44.736699 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:44.738178 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:44.738816 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:44.739089 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:44.740119 19384 leader_election.cc:288] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.740749 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:44.741079 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:44.740130 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:44.742137 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:44.740492 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:44.740458 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:44.742877 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:44.745388 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.743340 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:44.743498 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:44.747553 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.747645 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:44.747991 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.730265 19347 raft_consensus.cc:482] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:44.747133 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.749182 19347 raft_consensus.cc:3037] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:44.749342 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.750999 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.752590 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.755019 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.756183 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.756311 19347 raft_consensus.cc:504] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.758141 19347 leader_election.cc:288] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:44.760486 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.764905 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.767098 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:38:44.768625 18109 debug-util.cc:398] Leaking SignalData structure 0x7b08000e85e0 after lost signal to thread 16157
I20260807 03:38:44.768769 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:38:44.769553 18109 debug-util.cc:398] Leaking SignalData structure 0x7b0800124320 after lost signal to thread 17976
W20260807 03:38:44.770619 18109 debug-util.cc:398] Leaking SignalData structure 0x7b0800181c00 after lost signal to thread 18113
W20260807 03:38:44.771559 18109 debug-util.cc:398] Leaking SignalData structure 0x7b08002d9540 after lost signal to thread 18228
I20260807 03:38:44.772230 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:38:44.772558 18109 debug-util.cc:398] Leaking SignalData structure 0x7b08002cdb80 after lost signal to thread 18346
W20260807 03:38:44.773708 18109 debug-util.cc:398] Leaking SignalData structure 0x7b0800237f00 after lost signal to thread 19330
I20260807 03:38:44.774777 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:44.775161 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.781899 19375 raft_consensus.cc:482] T af8877c3fbba4ab18765832f1390921f P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:44.782004 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:44.782261 19361 raft_consensus.cc:482] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:44.782469 19375 raft_consensus.cc:504] T af8877c3fbba4ab18765832f1390921f P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.782739 19361 raft_consensus.cc:504] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.783188 18140 leader_election.cc:302] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:44.778285 19347 raft_consensus.cc:482] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:44.783933 19353 raft_consensus.cc:2781] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:44.784489 19361 leader_election.cc:288] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:44.784303 19347 raft_consensus.cc:504] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.781396 19390 raft_consensus.cc:482] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:44.785230 19390 raft_consensus.cc:504] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.786163 19347 leader_election.cc:288] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:44.787115 19390 leader_election.cc:288] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:44.787744 19375 leader_election.cc:288] T af8877c3fbba4ab18765832f1390921f P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:44.784324 19353 raft_consensus.cc:482] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:44.790771 19353 raft_consensus.cc:3037] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:44.780066 18027 leader_election.cc:302] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:44.793197 19364 raft_consensus.cc:2781] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:44.793604 19364 raft_consensus.cc:482] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:44.793982 19364 raft_consensus.cc:3037] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:44.798233 19353 raft_consensus.cc:504] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.800463 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.800971 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.801432 19364 raft_consensus.cc:504] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.802088 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.802666 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:44.808287 19364 leader_election.cc:288] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.808518 18248 leader_election.cc:302] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:44.809425 18140 leader_election.cc:302] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:44.809592 19347 raft_consensus.cc:2726] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.811923 18248 leader_election.cc:302] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:44.812175 19341 raft_consensus.cc:2781] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:44.812597 19341 raft_consensus.cc:482] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:44.812990 19390 raft_consensus.cc:2726] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.813133 19341 raft_consensus.cc:3037] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:44.814227 18140 leader_election.cc:302] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:44.812631 19353 leader_election.cc:288] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.815802 18248 leader_election.cc:302] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:44.819083 19347 raft_consensus.cc:2726] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.819537 19369 raft_consensus.cc:2781] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:44.820011 19369 raft_consensus.cc:482] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:44.820436 19369 raft_consensus.cc:3037] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:44.822203 19341 raft_consensus.cc:504] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.825492 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.826252 18248 leader_election.cc:302] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:44.826433 19341 leader_election.cc:288] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.827308 18248 leader_election.cc:302] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:44.827369 19369 raft_consensus.cc:504] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.828509 19347 raft_consensus.cc:2726] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.828389 18248 leader_election.cc:302] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:44.829181 19347 raft_consensus.cc:2726] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.829375 19369 leader_election.cc:288] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.829838 18248 leader_election.cc:302] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:44.830327 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.831174 18248 leader_election.cc:302] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:44.830480 19347 raft_consensus.cc:2726] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.832026 19375 raft_consensus.cc:2726] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.833485 18248 leader_election.cc:302] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:44.834846 19347 raft_consensus.cc:2726] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.835180 18027 leader_election.cc:302] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:38:44.835952 19352 raft_consensus.cc:2781] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:44.828369 19390 raft_consensus.cc:2726] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.836539 19352 raft_consensus.cc:482] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:44.837294 19352 raft_consensus.cc:3037] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:44.839286 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:44.840462 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:44.841590 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:44.842420 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:44.843417 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:44.835928 18248 leader_election.cc:302] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:44.834300 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.844522 19390 raft_consensus.cc:2726] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.844609 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:44.845642 18248 leader_election.cc:302] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:44.846225 19390 raft_consensus.cc:2726] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.846590 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:44.846351 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.847188 18248 leader_election.cc:302] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:44.847702 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:44.848351 19390 raft_consensus.cc:2726] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.850832 18027 leader_election.cc:302] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:44.851387 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.851509 18248 leader_election.cc:302] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:44.852216 19390 raft_consensus.cc:2726] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.852813 19364 raft_consensus.cc:2726] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.853356 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.853757 18027 leader_election.cc:302] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:44.854342 19364 raft_consensus.cc:2726] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.854624 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.855060 18248 leader_election.cc:302] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:44.855623 19390 raft_consensus.cc:2726] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.843967 19352 raft_consensus.cc:504] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.856285 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.857363 19352 leader_election.cc:288] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.857693 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.857806 18248 leader_election.cc:302] T af8877c3fbba4ab18765832f1390921f P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:44.857812 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.859483 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.860090 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.861056 18027 leader_election.cc:302] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:44.861332 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.861683 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.861770 19390 raft_consensus.cc:2726] T af8877c3fbba4ab18765832f1390921f P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.861999 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.862905 18248 leader_election.cc:302] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:44.863557 19384 raft_consensus.cc:2726] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.863605 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.864799 18027 leader_election.cc:302] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:44.865291 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.865939 19390 raft_consensus.cc:2726] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.866892 18248 leader_election.cc:302] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:44.867019 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.867508 19390 raft_consensus.cc:2726] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.868558 18248 leader_election.cc:302] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:44.868790 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.869201 19384 raft_consensus.cc:2726] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.870548 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:44.870963 18027 leader_election.cc:302] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:38:44.871611 19390 raft_consensus.cc:2726] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.872251 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:44.872849 19384 raft_consensus.cc:2781] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:44.861140 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.873230 19384 raft_consensus.cc:482] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:44.873286 18027 leader_election.cc:302] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:44.873674 19384 raft_consensus.cc:3037] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:44.874027 19352 raft_consensus.cc:2726] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.874648 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:44.875065 18027 leader_election.cc:302] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:44.876464 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.876667 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.878412 19352 raft_consensus.cc:2726] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.878383 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.879698 18027 leader_election.cc:302] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:44.881752 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.882313 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.882690 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.883023 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.884081 19352 raft_consensus.cc:2726] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.885274 18027 leader_election.cc:302] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:44.880962 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.886094 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.886754 19352 raft_consensus.cc:2726] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.885429 19384 raft_consensus.cc:504] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.888263 18027 leader_election.cc:302] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:38:44.889091 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.890072 19384 leader_election.cc:288] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.890447 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:44.893024 19352 raft_consensus.cc:2781] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:44.893473 19352 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:44.893857 19352 raft_consensus.cc:3037] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:44.896602 18027 leader_election.cc:302] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:44.897138 19384 raft_consensus.cc:2726] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.897456 18027 leader_election.cc:302] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:38:44.898083 19384 raft_consensus.cc:2781] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:44.898449 18027 leader_election.cc:302] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:44.898460 19384 raft_consensus.cc:482] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:44.899205 19364 raft_consensus.cc:2726] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.899554 18027 leader_election.cc:302] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:38:44.899796 19384 raft_consensus.cc:3037] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:44.900302 19364 raft_consensus.cc:2781] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:44.900660 18027 leader_election.cc:302] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:44.900683 19364 raft_consensus.cc:482] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:44.901283 19364 raft_consensus.cc:3037] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:44.903467 18141 leader_election.cc:302] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:44.905558 19369 raft_consensus.cc:2726] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.906533 19243 raft_consensus.cc:2726] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.906880 18141 leader_election.cc:302] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:44.907608 18027 leader_election.cc:302] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:38:44.908244 19335 raft_consensus.cc:2726] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.908900 19364 raft_consensus.cc:504] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.910125 19243 raft_consensus.cc:2781] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:44.910490 19243 raft_consensus.cc:482] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:44.910826 19243 raft_consensus.cc:3037] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:44.911338 19364 leader_election.cc:288] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.916975 19384 raft_consensus.cc:504] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.919411 19384 leader_election.cc:288] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.918548 19243 raft_consensus.cc:504] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.920418 18141 leader_election.cc:302] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:44.921191 19365 raft_consensus.cc:2726] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.921510 19243 leader_election.cc:288] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.900807 19352 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.922829 18027 leader_election.cc:302] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:44.923564 19243 raft_consensus.cc:2726] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.923921 18027 leader_election.cc:302] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:44.924975 18027 leader_election.cc:302] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:44.925457 18141 leader_election.cc:302] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:44.926004 18027 leader_election.cc:302] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:44.926321 19352 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.926352 19364 raft_consensus.cc:2726] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.925256 19384 raft_consensus.cc:2726] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.926975 19341 raft_consensus.cc:2726] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.927848 18141 leader_election.cc:302] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:44.928627 19341 raft_consensus.cc:2726] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.929548 18141 leader_election.cc:302] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:44.930222 19341 raft_consensus.cc:2726] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.931294 18141 leader_election.cc:302] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:44.931985 19341 raft_consensus.cc:2726] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.933041 18141 leader_election.cc:302] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:44.933746 19341 raft_consensus.cc:2726] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.934845 18141 leader_election.cc:302] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:44.935518 19341 raft_consensus.cc:2726] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.937573 18141 leader_election.cc:302] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:44.938323 19341 raft_consensus.cc:2726] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.939450 18141 leader_election.cc:302] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:44.940181 19341 raft_consensus.cc:2726] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.941267 18141 leader_election.cc:302] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:44.941975 19341 raft_consensus.cc:2726] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.943882 18141 leader_election.cc:302] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:44.944597 19341 raft_consensus.cc:2726] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.945744 18141 leader_election.cc:302] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:44.946406 19341 raft_consensus.cc:2726] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.947461 18141 leader_election.cc:302] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:44.948122 19341 raft_consensus.cc:2726] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.948493 18141 leader_election.cc:302] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:44.949353 19341 raft_consensus.cc:2726] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.950981 18141 leader_election.cc:302] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:44.951586 19341 raft_consensus.cc:2726] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.954552 18079 raft_consensus.cc:1226] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:44.954612 18141 leader_election.cc:302] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:44.955314 19341 raft_consensus.cc:2726] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.956377 19361 consensus_queue.cc:1046] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:44.956403 18141 leader_election.cc:302] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:44.957332 19361 raft_consensus.cc:3032] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:44.957695 19341 raft_consensus.cc:2726] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.957695 19361 raft_consensus.cc:729] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:44.959008 19361 consensus_queue.cc:260] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.959581 19344 raft_consensus.cc:2726] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.960451 19361 raft_consensus.cc:3037] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:44.960973 18027 leader_election.cc:302] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:38:44.961632 19243 raft_consensus.cc:2781] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:44.961921 19243 raft_consensus.cc:482] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:44.962318 19243 raft_consensus.cc:3037] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:44.962019 18309 consensus_queue.cc:237] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:38:44.963299 18141 leader_election.cc:302] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:44.963322 18027 leader_election.cc:302] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:38:44.969204 19384 raft_consensus.cc:2781] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:44.969487 19365 raft_consensus.cc:2726] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.969543 19384 raft_consensus.cc:482] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:44.970317 19384 raft_consensus.cc:3037] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:44.970454 18027 leader_election.cc:302] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:44.972191 19344 raft_consensus.cc:2726] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.973234 18027 leader_election.cc:302] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:44.974920 19344 raft_consensus.cc:2726] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.975835 18027 leader_election.cc:302] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:44.977556 19344 raft_consensus.cc:2726] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.977475 19384 raft_consensus.cc:504] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.979218 18027 leader_election.cc:302] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:44.977335 19243 raft_consensus.cc:504] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:44.982144 19384 leader_election.cc:288] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.984632 19243 leader_election.cc:288] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:44.985114 18184 raft_consensus.cc:1261] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:38:44.986707 19347 consensus_queue.cc:1035] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:44.987108 19344 raft_consensus.cc:2726] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.989396 18027 leader_election.cc:302] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:44.990410 18079 raft_consensus.cc:1261] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:38:44.992113 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:44.992327 19357 raft_consensus.cc:2726] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:44.993471 19361 consensus_queue.cc:1035] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:45.000149 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:45.002594 18027 leader_election.cc:302] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:45.003064 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:45.003613 18027 leader_election.cc:302] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:45.004079 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:45.006664 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:45.006736 19347 raft_consensus.cc:2932] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:38:45.010645 18193 raft_consensus.cc:2932] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:38:45.011680 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:45.015389 18309 raft_consensus.cc:2454] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 2.
I20260807 03:38:45.016240 19344 raft_consensus.cc:2726] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:45.016522 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:45.016899 19344 raft_consensus.cc:2726] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:45.017994 18027 leader_election.cc:302] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:45.018450 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:45.018621 19344 raft_consensus.cc:2726] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:45.018677 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:45.019887 18027 leader_election.cc:302] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:45.020673 19243 raft_consensus.cc:2726] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:45.021538 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:45.021818 18027 leader_election.cc:302] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:45.022508 19243 raft_consensus.cc:2726] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:45.022871 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:45.023067 18081 raft_consensus.cc:1226] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:45.023614 18027 leader_election.cc:302] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:45.024315 19243 raft_consensus.cc:2726] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:45.025286 18309 raft_consensus.cc:1226] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:45.026567 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:45.027086 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:45.028470 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:45.028616 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:45.030444 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:45.031090 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:45.025221 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:45.045904 18081 raft_consensus.cc:1226] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:45.058151 18081 raft_consensus.cc:1226] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:45.065958 18027 leader_election.cc:302] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:45.070030 19344 raft_consensus.cc:2726] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 03:38:45.071408 18027 leader_election.cc:302] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:38:45.072726 19369 consensus_queue.cc:1046] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 03:38:45.073931 19369 consensus_queue.cc:1046] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:45.075129 19353 raft_consensus.cc:3032] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:45.075469 19353 raft_consensus.cc:729] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:45.076292 19353 consensus_queue.cc:260] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:45.077440 19353 raft_consensus.cc:3037] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:45.078329 19384 raft_consensus.cc:2781] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:38:45.078831 19384 raft_consensus.cc:686] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 2 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:38:45.079684 19384 consensus_queue.cc:237] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:45.080997 19392 raft_consensus.cc:3032] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:45.081377 19392 raft_consensus.cc:729] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:45.082283 19392 consensus_queue.cc:260] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:45.085938 18027 leader_election.cc:302] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:45.087998 19243 raft_consensus.cc:2726] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 03:38:45.090428 18075 raft_consensus.cc:2932] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:38:45.092358 19392 raft_consensus.cc:3037] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:45.096978 19369 consensus_queue.cc:1046] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:45.098135 19341 raft_consensus.cc:3032] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:45.098521 19341 raft_consensus.cc:729] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:45.099406 19341 consensus_queue.cc:260] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:45.101338 19341 raft_consensus.cc:3037] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:45.106534 18027 leader_election.cc:302] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:45.107512 19243 raft_consensus.cc:2726] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 03:38:45.109460 19384 consensus_queue.cc:1046] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:45.110726 19243 raft_consensus.cc:3032] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:45.111084 19243 raft_consensus.cc:729] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:38:45.111951 19243 consensus_queue.cc:260] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:45.113504 19243 raft_consensus.cc:3037] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:45.119040 17912 catalog_manager.cc:5495] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 1 to 2, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 2 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:45.123136 18075 raft_consensus.cc:1226] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3. Ops: []
I20260807 03:38:45.131996 18027 leader_election.cc:302] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:45.132793 19384 raft_consensus.cc:2726] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 03:38:45.141319 18073 raft_consensus.cc:1226] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:45.151044 19369 consensus_queue.cc:1046] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:45.152185 18190 raft_consensus.cc:1226] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:45.152885 18027 leader_election.cc:302] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:45.153350 19335 raft_consensus.cc:3032] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 03:38:45.153769 19335 raft_consensus.cc:729] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:45.153645 19347 consensus_queue.cc:1046] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 03:38:45.154273 18027 leader_election.cc:302] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:45.154981 19384 raft_consensus.cc:2726] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 03:38:45.155095 19401 raft_consensus.cc:3032] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:45.154680 19335 consensus_queue.cc:260] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:45.155638 19384 raft_consensus.cc:2726] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 03:38:45.155650 19401 raft_consensus.cc:729] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
W20260807 03:38:45.156248 19372 consensus_queue.cc:1159] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "66a36e1ac8e945e893668e282d394c49" responder_term: 2 status { last_received { term: 2 index: 1 } last_committed_idx: 1 last_received_current_leader { term: 2 index: 1 } }
I20260807 03:38:45.156677 18027 leader_election.cc:302] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:45.157429 19401 consensus_queue.cc:260] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:45.158026 19376 consensus_queue.cc:1046] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:45.161062 19376 raft_consensus.cc:3032] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:45.161382 19376 raft_consensus.cc:729] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:45.162186 19376 consensus_queue.cc:260] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:45.163568 19376 raft_consensus.cc:3037] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:45.164974 17920 catalog_manager.cc:5495] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: config changed from index -1 to 2, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New cstate: current_term: 1 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:38:45.159112 19335 raft_consensus.cc:3037] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:45.167645 19384 raft_consensus.cc:2726] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 03:38:45.167433 18027 leader_election.cc:302] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
W20260807 03:38:45.169363 18246 consensus_peers.cc:487] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 -> Peer 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381): Couldn't send request to peer 5c9b5062bdfa46af9396f1a85025a331. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 071a939ead594d38851e3b554bfc4d24. This is attempt 1: this message will repeat every 5th retry.
I20260807 03:38:45.170279 18293 consensus_queue.cc:237] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:38:45.168915 19384 raft_consensus.cc:2726] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 03:38:45.173596 19401 raft_consensus.cc:3037] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:45.195024 18193 raft_consensus.cc:1261] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:38:45.196451 19401 consensus_queue.cc:1035] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260807 03:38:45.205825 18246 consensus_peers.cc:487] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 -> Peer 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381): Couldn't send request to peer 5c9b5062bdfa46af9396f1a85025a331. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9a59a83a64df475ea15dd572f84024bd. This is attempt 1: this message will repeat every 5th retry.
I20260807 03:38:45.206251 18079 raft_consensus.cc:1261] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:38:45.210271 18187 raft_consensus.cc:1226] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:45.207726 19401 consensus_queue.cc:1035] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260807 03:38:45.211580 19347 consensus_queue.cc:1046] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:45.212527 19390 raft_consensus.cc:3032] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:45.212874 19390 raft_consensus.cc:729] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:45.213725 19390 consensus_queue.cc:260] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:45.215062 19390 raft_consensus.cc:3037] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:45.229137 18293 raft_consensus.cc:1226] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:45.233400 19378 consensus_queue.cc:1046] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 03:38:45.234601 19243 raft_consensus.cc:3032] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:45.234948 19243 raft_consensus.cc:729] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:38:45.235771 19243 consensus_queue.cc:260] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:45.235596 19347 raft_consensus.cc:2932] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:38:45.237114 19243 raft_consensus.cc:3037] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:45.237004 18193 raft_consensus.cc:2932] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:38:45.242408 18187 raft_consensus.cc:1226] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:45.245186 19401 consensus_queue.cc:1046] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 03:38:45.247447 19401 raft_consensus.cc:3032] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:45.247831 19401 raft_consensus.cc:729] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:45.249575 19401 consensus_queue.cc:260] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:45.251379 19401 raft_consensus.cc:3037] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:45.274294 18079 raft_consensus.cc:1226] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:45.273954 18075 raft_consensus.cc:2932] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:38:45.276909 19335 consensus_queue.cc:1046] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:45.277895 19376 raft_consensus.cc:3032] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:45.278290 19376 raft_consensus.cc:729] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:45.279104 19376 consensus_queue.cc:260] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:45.280459 19376 raft_consensus.cc:3037] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:45.302852 17920 catalog_manager.cc:5495] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 reported cstate change: config changed from index -1 to 2, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New cstate: current_term: 1 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:45.309952 18193 consensus_queue.cc:237] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:38:45.329094 18293 raft_consensus.cc:1261] T af8877c3fbba4ab18765832f1390921f P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:38:45.330749 19353 consensus_queue.cc:1035] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 03:38:45.351722 18079 raft_consensus.cc:1261] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:38:45.353957 19335 consensus_queue.cc:1035] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:45.367388 19335 raft_consensus.cc:2932] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:38:45.368862 18295 raft_consensus.cc:2932] T af8877c3fbba4ab18765832f1390921f P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
W20260807 03:38:45.383936 18138 consensus_peers.cc:487] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 -> Peer 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381): Couldn't send request to peer 5c9b5062bdfa46af9396f1a85025a331. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: af8877c3fbba4ab18765832f1390921f. This is attempt 1: this message will repeat every 5th retry.
I20260807 03:38:45.395582 17912 catalog_manager.cc:5495] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: config changed from index -1 to 2, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New cstate: current_term: 1 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:45.396531 18293 consensus_queue.cc:237] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:38:45.412014 18187 raft_consensus.cc:1261] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:38:45.413946 19401 consensus_queue.cc:1035] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:45.428082 18073 raft_consensus.cc:2932] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:38:45.440210 18073 raft_consensus.cc:1261] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260807 03:38:45.442126 18246 consensus_peers.cc:487] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 -> Peer 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381): Couldn't send request to peer 5c9b5062bdfa46af9396f1a85025a331. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1b214306f53244f8b6ff2a1a9019abd1. This is attempt 1: this message will repeat every 5th retry.
I20260807 03:38:45.442162 19390 consensus_queue.cc:1035] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:45.446730 18079 raft_consensus.cc:1226] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:45.448446 19353 consensus_queue.cc:1046] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:45.447710 19390 raft_consensus.cc:2932] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:38:45.449410 19353 raft_consensus.cc:3032] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:45.449766 19353 raft_consensus.cc:729] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:45.450493 19353 consensus_queue.cc:260] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:45.451727 19353 raft_consensus.cc:3037] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:45.456696 18187 raft_consensus.cc:2932] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:38:45.477155 17920 catalog_manager.cc:5495] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 reported cstate change: config changed from index -1 to 2, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New cstate: current_term: 1 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:45.487811 18187 raft_consensus.cc:3037] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:45.496114 18187 raft_consensus.cc:1261] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:38:45.501492 19378 consensus_queue.cc:1035] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:45.520349 18295 raft_consensus.cc:1261] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:38:45.522716 18075 raft_consensus.cc:2932] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:38:45.537822 18079 raft_consensus.cc:1226] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:45.539909 19384 consensus_queue.cc:1035] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20260807 03:38:45.558194 19353 consensus_queue.cc:1046] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20260807 03:38:45.564519 19426 ts_tablet_manager.cc:927] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331: Initiating tablet copy from peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:45.585407 19392 raft_consensus.cc:3032] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:45.585775 19392 raft_consensus.cc:729] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:45.586513 19392 consensus_queue.cc:260] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:45.587721 19392 raft_consensus.cc:3037] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:45.614928 19426 tablet_copy_client.cc:316] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Beginning tablet copy session from remote peer at address 127.15.198.131:33021
I20260807 03:38:45.679950 18326 tablet_copy_service.cc:140] P 66a36e1ac8e945e893668e282d394c49: Received BeginTabletCopySession request for tablet 071a939ead594d38851e3b554bfc4d24 from peer 5c9b5062bdfa46af9396f1a85025a331 ({username='slave'} at 127.0.0.1:37190)
I20260807 03:38:45.680677 18326 tablet_copy_service.cc:161] P 66a36e1ac8e945e893668e282d394c49: Beginning new tablet copy session on tablet 071a939ead594d38851e3b554bfc4d24 from peer 5c9b5062bdfa46af9396f1a85025a331 at {username='slave'} at 127.0.0.1:37190: session id = 5c9b5062bdfa46af9396f1a85025a331-071a939ead594d38851e3b554bfc4d24
I20260807 03:38:45.691213 18326 tablet_copy_source_session.cc:205] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 03:38:45.695184 19426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 071a939ead594d38851e3b554bfc4d24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:45.716756 19426 tablet_copy_client.cc:791] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Starting download of 0 data blocks...
I20260807 03:38:45.718863 19426 tablet_copy_client.cc:655] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Starting download of 1 WAL segments...
I20260807 03:38:45.724713 19426 tablet_copy_client.cc:531] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 03:38:45.734426 19426 tablet_bootstrap.cc:492] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331: Bootstrap starting.
I20260807 03:38:45.750866 19436 ts_tablet_manager.cc:927] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331: Initiating tablet copy from peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:45.769148 19436 tablet_copy_client.cc:316] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Beginning tablet copy session from remote peer at address 127.15.198.130:37057
I20260807 03:38:45.776988 19426 tablet_bootstrap.cc:492] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 03:38:45.777848 19426 tablet_bootstrap.cc:492] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331: Bootstrap complete.
I20260807 03:38:45.778380 19426 ts_tablet_manager.cc:1397] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331: Time spent bootstrapping tablet: real 0.044s user 0.023s sys 0.015s
I20260807 03:38:45.781248 19426 raft_consensus.cc:348] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:38:45.782194 19426 raft_consensus.cc:729] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5c9b5062bdfa46af9396f1a85025a331, State: Initialized, Role: LEARNER
I20260807 03:38:45.782892 19426 consensus_queue.cc:260] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:38:45.786125 18207 tablet_copy_service.cc:140] P 3c4bceb836db4536a0f255fdf86abc02: Received BeginTabletCopySession request for tablet af8877c3fbba4ab18765832f1390921f from peer 5c9b5062bdfa46af9396f1a85025a331 ({username='slave'} at 127.0.0.1:50060)
I20260807 03:38:45.786914 18207 tablet_copy_service.cc:161] P 3c4bceb836db4536a0f255fdf86abc02: Beginning new tablet copy session on tablet af8877c3fbba4ab18765832f1390921f from peer 5c9b5062bdfa46af9396f1a85025a331 at {username='slave'} at 127.0.0.1:50060: session id = 5c9b5062bdfa46af9396f1a85025a331-af8877c3fbba4ab18765832f1390921f
I20260807 03:38:45.792464 19426 ts_tablet_manager.cc:1428] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331: Time spent starting tablet: real 0.014s user 0.004s sys 0.011s
I20260807 03:38:45.794502 18326 tablet_copy_service.cc:342] P 66a36e1ac8e945e893668e282d394c49: Request end of tablet copy session 5c9b5062bdfa46af9396f1a85025a331-071a939ead594d38851e3b554bfc4d24 received from {username='slave'} at 127.0.0.1:37190
I20260807 03:38:45.794943 18326 tablet_copy_service.cc:433] P 66a36e1ac8e945e893668e282d394c49: ending tablet copy session 5c9b5062bdfa46af9396f1a85025a331-071a939ead594d38851e3b554bfc4d24 on tablet 071a939ead594d38851e3b554bfc4d24 with peer 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:38:45.808550 18207 tablet_copy_source_session.cc:205] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 03:38:45.811771 19426 ts_tablet_manager.cc:927] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331: Initiating tablet copy from peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:45.812048 19436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af8877c3fbba4ab18765832f1390921f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:45.860427 19436 tablet_copy_client.cc:791] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Starting download of 0 data blocks...
I20260807 03:38:45.861052 19436 tablet_copy_client.cc:655] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Starting download of 1 WAL segments...
I20260807 03:38:45.865424 19436 tablet_copy_client.cc:531] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 03:38:45.873167 19443 ts_tablet_manager.cc:927] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331: Initiating tablet copy from peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:45.873503 19436 tablet_bootstrap.cc:492] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331: Bootstrap starting.
I20260807 03:38:45.880587 19443 tablet_copy_client.cc:316] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Beginning tablet copy session from remote peer at address 127.15.198.131:33021
I20260807 03:38:45.883684 18326 tablet_copy_service.cc:140] P 66a36e1ac8e945e893668e282d394c49: Received BeginTabletCopySession request for tablet 1b214306f53244f8b6ff2a1a9019abd1 from peer 5c9b5062bdfa46af9396f1a85025a331 ({username='slave'} at 127.0.0.1:37190)
I20260807 03:38:45.884316 18326 tablet_copy_service.cc:161] P 66a36e1ac8e945e893668e282d394c49: Beginning new tablet copy session on tablet 1b214306f53244f8b6ff2a1a9019abd1 from peer 5c9b5062bdfa46af9396f1a85025a331 at {username='slave'} at 127.0.0.1:37190: session id = 5c9b5062bdfa46af9396f1a85025a331-1b214306f53244f8b6ff2a1a9019abd1
I20260807 03:38:45.885201 19426 tablet_copy_client.cc:316] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Beginning tablet copy session from remote peer at address 127.15.198.131:33021
I20260807 03:38:45.886904 18325 tablet_copy_service.cc:140] P 66a36e1ac8e945e893668e282d394c49: Received BeginTabletCopySession request for tablet 9a59a83a64df475ea15dd572f84024bd from peer 5c9b5062bdfa46af9396f1a85025a331 ({username='slave'} at 127.0.0.1:37190)
I20260807 03:38:45.887382 18325 tablet_copy_service.cc:161] P 66a36e1ac8e945e893668e282d394c49: Beginning new tablet copy session on tablet 9a59a83a64df475ea15dd572f84024bd from peer 5c9b5062bdfa46af9396f1a85025a331 at {username='slave'} at 127.0.0.1:37190: session id = 5c9b5062bdfa46af9396f1a85025a331-9a59a83a64df475ea15dd572f84024bd
I20260807 03:38:45.893996 19297 raft_consensus.cc:1203] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 03:38:45.929181 18326 tablet_copy_source_session.cc:205] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 03:38:45.930745 18325 tablet_copy_source_session.cc:205] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 03:38:45.936401 19443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b214306f53244f8b6ff2a1a9019abd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:45.947204 19426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a59a83a64df475ea15dd572f84024bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:38:45.974176 19443 tablet_copy_client.cc:791] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Starting download of 0 data blocks...
I20260807 03:38:45.974817 19443 tablet_copy_client.cc:655] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Starting download of 1 WAL segments...
I20260807 03:38:45.984869 19443 tablet_copy_client.cc:531] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 03:38:45.996876 19443 tablet_bootstrap.cc:492] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331: Bootstrap starting.
I20260807 03:38:46.041993 19443 tablet_bootstrap.cc:492] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 03:38:46.043412 19426 tablet_copy_client.cc:791] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Starting download of 0 data blocks...
I20260807 03:38:46.044309 19426 tablet_copy_client.cc:655] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Starting download of 1 WAL segments...
I20260807 03:38:46.044548 19443 tablet_bootstrap.cc:492] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331: Bootstrap complete.
I20260807 03:38:46.045295 19443 ts_tablet_manager.cc:1397] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331: Time spent bootstrapping tablet: real 0.049s user 0.015s sys 0.009s
I20260807 03:38:46.048170 19443 raft_consensus.cc:348] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:38:46.049062 19443 raft_consensus.cc:729] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5c9b5062bdfa46af9396f1a85025a331, State: Initialized, Role: LEARNER
I20260807 03:38:46.049427 19426 tablet_copy_client.cc:531] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 03:38:46.049772 19443 consensus_queue.cc:260] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:38:46.054996 19443 ts_tablet_manager.cc:1428] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260807 03:38:46.056499 18325 tablet_copy_service.cc:342] P 66a36e1ac8e945e893668e282d394c49: Request end of tablet copy session 5c9b5062bdfa46af9396f1a85025a331-1b214306f53244f8b6ff2a1a9019abd1 received from {username='slave'} at 127.0.0.1:37190
I20260807 03:38:46.056870 18325 tablet_copy_service.cc:433] P 66a36e1ac8e945e893668e282d394c49: ending tablet copy session 5c9b5062bdfa46af9396f1a85025a331-1b214306f53244f8b6ff2a1a9019abd1 on tablet 1b214306f53244f8b6ff2a1a9019abd1 with peer 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:38:46.067364 19426 tablet_bootstrap.cc:492] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331: Bootstrap starting.
I20260807 03:38:46.067957 19436 tablet_bootstrap.cc:492] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 03:38:46.068809 19436 tablet_bootstrap.cc:492] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331: Bootstrap complete.
I20260807 03:38:46.069450 19436 ts_tablet_manager.cc:1397] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331: Time spent bootstrapping tablet: real 0.196s user 0.027s sys 0.058s
I20260807 03:38:46.072064 19436 raft_consensus.cc:348] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:38:46.072932 19436 raft_consensus.cc:729] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5c9b5062bdfa46af9396f1a85025a331, State: Initialized, Role: LEARNER
I20260807 03:38:46.074120 19436 consensus_queue.cc:260] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:38:46.082770 19436 ts_tablet_manager.cc:1428] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331: Time spent starting tablet: real 0.013s user 0.005s sys 0.002s
I20260807 03:38:46.089800 18207 tablet_copy_service.cc:342] P 3c4bceb836db4536a0f255fdf86abc02: Request end of tablet copy session 5c9b5062bdfa46af9396f1a85025a331-af8877c3fbba4ab18765832f1390921f received from {username='slave'} at 127.0.0.1:50060
I20260807 03:38:46.090183 18207 tablet_copy_service.cc:433] P 3c4bceb836db4536a0f255fdf86abc02: ending tablet copy session 5c9b5062bdfa46af9396f1a85025a331-af8877c3fbba4ab18765832f1390921f on tablet af8877c3fbba4ab18765832f1390921f with peer 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:38:46.117252 19426 tablet_bootstrap.cc:492] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 03:38:46.118167 19426 tablet_bootstrap.cc:492] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331: Bootstrap complete.
I20260807 03:38:46.118885 19426 ts_tablet_manager.cc:1397] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331: Time spent bootstrapping tablet: real 0.052s user 0.015s sys 0.013s
I20260807 03:38:46.121731 19426 raft_consensus.cc:348] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:38:46.122582 19426 raft_consensus.cc:729] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5c9b5062bdfa46af9396f1a85025a331, State: Initialized, Role: LEARNER
I20260807 03:38:46.123198 19426 consensus_queue.cc:260] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:38:46.125624 19426 ts_tablet_manager.cc:1428] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260807 03:38:46.127116 18325 tablet_copy_service.cc:342] P 66a36e1ac8e945e893668e282d394c49: Request end of tablet copy session 5c9b5062bdfa46af9396f1a85025a331-9a59a83a64df475ea15dd572f84024bd received from {username='slave'} at 127.0.0.1:37190
I20260807 03:38:46.127449 18325 tablet_copy_service.cc:433] P 66a36e1ac8e945e893668e282d394c49: ending tablet copy session 5c9b5062bdfa46af9396f1a85025a331-9a59a83a64df475ea15dd572f84024bd on tablet 9a59a83a64df475ea15dd572f84024bd with peer 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:38:46.198356 19335 raft_consensus.cc:482] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:46.198977 19335 raft_consensus.cc:504] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:46.201366 19335 leader_election.cc:288] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:46.202366 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:46.203001 18079 raft_consensus.cc:2454] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:38:46.201862 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:46.204077 18140 leader_election.cc:302] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:46.204325 18300 raft_consensus.cc:2379] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 2: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:38:46.204811 19335 raft_consensus.cc:2781] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:46.205838 19335 raft_consensus.cc:482] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:46.206281 19335 raft_consensus.cc:3037] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:46.213573 19335 raft_consensus.cc:504] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:46.217953 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:46.219117 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:46.219973 18300 raft_consensus.cc:2379] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 2: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:38:46.222705 19335 leader_election.cc:288] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:46.225378 18075 raft_consensus.cc:2454] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:38:46.232532 19392 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:46.233021 19392 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:46.233525 19297 raft_consensus.cc:1203] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 03:38:46.235795 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:46.237186 19392 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:46.254173 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:46.254777 18075 raft_consensus.cc:2454] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:38:46.257810 18140 leader_election.cc:302] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:46.258939 19392 raft_consensus.cc:2781] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:38:46.259490 19392 raft_consensus.cc:686] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 2 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:46.258912 18140 leader_election.cc:302] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:46.260564 19392 consensus_queue.cc:237] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:46.262233 19454 raft_consensus.cc:2781] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:46.262567 19454 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:46.263449 19454 raft_consensus.cc:3037] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:46.277748 19454 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:46.284533 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:46.285141 18073 raft_consensus.cc:3037] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:46.288838 19454 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:46.289757 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:46.291712 18073 raft_consensus.cc:2454] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:38:46.292878 18140 leader_election.cc:302] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:46.294137 19454 raft_consensus.cc:2781] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:38:46.294971 19454 raft_consensus.cc:686] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 3 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:46.301124 17920 catalog_manager.cc:5495] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 1 to 2, leader changed from d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 2 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:46.295980 19454 consensus_queue.cc:237] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:46.356499 19392 raft_consensus.cc:482] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:46.358450 19392 raft_consensus.cc:504] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:46.382894 19297 raft_consensus.cc:1203] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 03:38:46.389382 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:46.403494 18075 raft_consensus.cc:2454] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:38:46.405472 18140 leader_election.cc:302] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:46.406817 19335 raft_consensus.cc:2781] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:46.407361 19335 raft_consensus.cc:482] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:46.407804 19335 raft_consensus.cc:3037] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:46.445448 19297 raft_consensus.cc:1203] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 03:38:46.447324 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:46.391232 19392 leader_election.cc:288] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:46.462502 19335 raft_consensus.cc:504] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:46.467782 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:46.469264 19456 raft_consensus.cc:482] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:46.469120 17915 catalog_manager.cc:5495] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:46.469951 19456 raft_consensus.cc:504] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:46.472251 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:46.472779 19456 leader_election.cc:288] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:46.475281 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:46.476253 18075 raft_consensus.cc:2454] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:38:46.477756 18140 leader_election.cc:302] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:46.479378 19456 raft_consensus.cc:2781] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:38:46.479851 19456 raft_consensus.cc:482] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:46.480299 19456 raft_consensus.cc:3037] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:38:46.483088 19347 raft_consensus.cc:482] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:46.483548 19390 raft_consensus.cc:1050] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49: attempting to promote NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 to VOTER
I20260807 03:38:46.472810 18073 raft_consensus.cc:3037] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:46.491037 18073 raft_consensus.cc:2454] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:38:46.483729 19347 raft_consensus.cc:504] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:46.466850 19335 leader_election.cc:288] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:46.494774 19347 leader_election.cc:288] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:46.494984 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:46.494769 19335 raft_consensus.cc:482] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:46.496192 19392 raft_consensus.cc:482] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:46.496061 19335 raft_consensus.cc:504] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:46.496857 19392 raft_consensus.cc:504] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:46.499465 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:46.500133 19392 leader_election.cc:288] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:46.501740 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:46.502182 19388 raft_consensus.cc:482] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:46.502599 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:46.502803 19388 raft_consensus.cc:504] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:46.503394 18190 raft_consensus.cc:2454] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
I20260807 03:38:46.503702 19335 leader_election.cc:288] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:46.504575 19388 leader_election.cc:288] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:46.505218 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:46.506536 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:46.507509 18188 raft_consensus.cc:2454] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
I20260807 03:38:46.506582 18249 leader_election.cc:302] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:46.509966 19347 raft_consensus.cc:2781] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:46.510398 19347 raft_consensus.cc:482] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:46.510771 19347 raft_consensus.cc:3037] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:46.514336 18140 leader_election.cc:302] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:46.517167 19454 raft_consensus.cc:2781] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:38:46.517777 19454 raft_consensus.cc:686] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:46.518004 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:46.519008 18075 raft_consensus.cc:2454] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:38:46.519541 19454 consensus_queue.cc:237] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:46.521552 18249 leader_election.cc:302] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:46.523178 19472 raft_consensus.cc:2781] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:46.523702 19472 raft_consensus.cc:482] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:46.524104 19472 raft_consensus.cc:3037] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:46.524495 19378 raft_consensus.cc:482] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:46.525205 19378 raft_consensus.cc:504] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:46.529386 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:46.530228 18300 raft_consensus.cc:2454] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 2.
I20260807 03:38:46.477856 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:46.533375 19378 leader_election.cc:288] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:46.538516 18027 leader_election.cc:302] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:46.539844 19453 raft_consensus.cc:2781] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:46.540288 19453 raft_consensus.cc:482] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:46.540714 19453 raft_consensus.cc:3037] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:46.550670 19344 raft_consensus.cc:482] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:46.523077 19390 consensus_queue.cc:237] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:38:46.555958 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:46.556285 19472 raft_consensus.cc:504] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:46.551807 19344 raft_consensus.cc:504] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:46.595207 18073 raft_consensus.cc:2454] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:38:46.599555 18140 leader_election.cc:302] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:46.611058 19471 raft_consensus.cc:482] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:46.612303 19471 raft_consensus.cc:504] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:46.616075 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:46.622087 19478 raft_consensus.cc:482] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:46.623680 19478 raft_consensus.cc:504] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:46.630919 19472 leader_election.cc:288] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:46.640349 18140 leader_election.cc:302] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:46.657526 19378 raft_consensus.cc:482] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:46.658442 19378 raft_consensus.cc:504] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:46.660622 19483 raft_consensus.cc:482] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:46.662122 19483 raft_consensus.cc:504] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:46.664394 19473 raft_consensus.cc:2781] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:46.663934 19347 raft_consensus.cc:504] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:46.665086 19473 raft_consensus.cc:482] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:46.665763 19473 raft_consensus.cc:3037] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:46.666714 19378 leader_election.cc:288] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:46.667347 19468 raft_consensus.cc:482] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:46.669108 19469 raft_consensus.cc:482] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:46.650179 19472 raft_consensus.cc:482] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:46.699002 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:46.703258 18079 raft_consensus.cc:3037] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 3
I20260807 03:38:46.692932 19478 leader_election.cc:288] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:46.688490 19344 leader_election.cc:288] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:46.698601 19335 raft_consensus.cc:2781] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:46.714146 19485 raft_consensus.cc:482] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:46.706871 19468 raft_consensus.cc:504] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:46.708058 19469 raft_consensus.cc:504] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:46.713085 19335 raft_consensus.cc:482] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:46.731770 19335 raft_consensus.cc:3037] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:46.715406 19485 raft_consensus.cc:504] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:46.709583 19472 raft_consensus.cc:504] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:46.744923 19456 raft_consensus.cc:504] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:46.755878 19468 leader_election.cc:288] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:46.771242 19469 leader_election.cc:288] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:46.772495 19453 raft_consensus.cc:504] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:46.761040 19347 leader_election.cc:288] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:46.766112 19471 leader_election.cc:288] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:46.790614 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:46.796324 18188 raft_consensus.cc:3037] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:46.802094 18079 raft_consensus.cc:2454] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:38:46.805882 19483 leader_election.cc:288] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:46.793226 19456 leader_election.cc:288] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:46.786269 19478 raft_consensus.cc:482] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:46.813354 19478 raft_consensus.cc:504] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:46.815794 19472 leader_election.cc:288] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:46.816203 19485 leader_election.cc:288] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:46.840416 19472 raft_consensus.cc:482] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:46.821521 19496 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:46.841667 19472 raft_consensus.cc:504] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:46.842162 19494 raft_consensus.cc:1050] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02: attempting to promote NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 to VOTER
I20260807 03:38:46.843429 19496 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:46.845726 19472 leader_election.cc:288] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:46.819393 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:46.848661 19496 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:46.821430 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:46.849747 19453 leader_election.cc:288] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:46.850163 18193 raft_consensus.cc:2454] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
I20260807 03:38:46.849272 18190 raft_consensus.cc:3037] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:46.852880 19469 raft_consensus.cc:482] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:46.853756 19469 raft_consensus.cc:504] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:46.847456 19494 consensus_queue.cc:237] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:38:46.857427 19469 leader_election.cc:288] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:46.857283 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:46.857769 19478 leader_election.cc:288] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:46.858614 18187 raft_consensus.cc:2454] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
W20260807 03:38:46.865363 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 225203us
I20260807 03:38:46.865501 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:46.872889 19499 raft_consensus.cc:482] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:46.873999 19499 raft_consensus.cc:504] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:46.878461 19499 leader_election.cc:288] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:46.884634 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:46.888881 18310 raft_consensus.cc:3037] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 4
I20260807 03:38:46.889241 18248 leader_election.cc:302] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:46.891288 19496 raft_consensus.cc:2781] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:38:46.892321 19496 raft_consensus.cc:686] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:46.894168 19496 consensus_queue.cc:237] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:46.900568 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:46.901521 18073 raft_consensus.cc:3037] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 3
I20260807 03:38:46.902623 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:46.903980 18081 raft_consensus.cc:2379] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 2: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:38:46.904510 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:46.906167 18072 raft_consensus.cc:2379] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 2: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:38:46.917141 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:46.918126 18184 raft_consensus.cc:1261] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 03:38:46.918198 18309 raft_consensus.cc:2454] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:38:46.920456 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:46.921854 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:46.925796 19473 raft_consensus.cc:504] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:46.927564 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:46.928769 18293 raft_consensus.cc:2454] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:38:46.929078 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:46.930437 18072 raft_consensus.cc:2379] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 2: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:38:46.930647 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:46.931907 18075 raft_consensus.cc:2354] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 2. Current term is 3.
I20260807 03:38:46.932140 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:46.932538 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:46.933089 18080 raft_consensus.cc:1261] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 03:38:46.934202 19473 leader_election.cc:288] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:46.933804 18071 raft_consensus.cc:2454] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
I20260807 03:38:46.933638 18188 raft_consensus.cc:2454] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:38:46.929095 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:46.937470 18073 raft_consensus.cc:2454] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:38:46.937930 18082 raft_consensus.cc:2379] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 3: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:38:46.927968 18185 raft_consensus.cc:2354] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 2. Current term is 3.
I20260807 03:38:46.941358 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:46.942272 18293 raft_consensus.cc:2454] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 2.
I20260807 03:38:46.943668 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:46.947710 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:46.948670 18300 raft_consensus.cc:3037] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:46.933610 18079 raft_consensus.cc:2379] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 2: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:38:46.977751 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:46.985110 19297 raft_consensus.cc:1261] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 LEARNER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 03:38:46.907342 17915 catalog_manager.cc:5495] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:46.988432 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:46.989447 18080 raft_consensus.cc:3037] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:38:46.993014 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:46.994403 18189 raft_consensus.cc:2354] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 3. Current term is 4.
I20260807 03:38:46.972536 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:46.999568 18297 raft_consensus.cc:2354] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 2. Current term is 3.
I20260807 03:38:47.006676 18073 raft_consensus.cc:2354] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 2. Current term is 3.
I20260807 03:38:47.013355 18190 raft_consensus.cc:2454] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:38:47.011911 19478 raft_consensus.cc:482] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:47.014727 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:47.022938 18300 raft_consensus.cc:2454] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3.
I20260807 03:38:47.027835 19492 raft_consensus.cc:482] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:47.028959 19492 raft_consensus.cc:504] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:47.032554 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:47.035360 19478 raft_consensus.cc:504] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:47.036659 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:47.037351 19492 leader_election.cc:288] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:47.037963 18184 raft_consensus.cc:2454] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
I20260807 03:38:47.039237 19478 leader_election.cc:288] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:47.041625 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:46.855933 19335 raft_consensus.cc:504] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:47.015738 18310 raft_consensus.cc:2454] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:38:47.044960 18193 raft_consensus.cc:2354] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 2. Current term is 3.
I20260807 03:38:47.046463 19335 leader_election.cc:288] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:47.052917 18071 raft_consensus.cc:2454] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:47.053855 18141 leader_election.cc:302] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:47.057888 18080 raft_consensus.cc:2454] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:38:47.067121 18027 leader_election.cc:302] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:47.068490 19453 raft_consensus.cc:2781] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:47.069635 19335 raft_consensus.cc:2781] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:47.071615 19453 raft_consensus.cc:482] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:47.072582 19453 raft_consensus.cc:3037] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:47.071907 19335 raft_consensus.cc:482] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:47.077564 18249 leader_election.cc:302] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:47.078271 19335 raft_consensus.cc:3037] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:47.081292 19485 raft_consensus.cc:2781] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:47.081946 19485 raft_consensus.cc:482] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:47.082628 19485 raft_consensus.cc:3037] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:47.088384 19478 raft_consensus.cc:482] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:47.090081 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:47.091045 18310 raft_consensus.cc:3037] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 3
I20260807 03:38:47.091241 19472 raft_consensus.cc:482] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:47.093039 18297 raft_consensus.cc:3037] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 3
I20260807 03:38:47.093623 19472 raft_consensus.cc:504] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:47.088513 18249 leader_election.cc:302] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:47.089516 19478 raft_consensus.cc:504] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:47.099004 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:47.111474 18300 raft_consensus.cc:3037] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 3
I20260807 03:38:47.099808 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:47.100971 19497 raft_consensus.cc:2781] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:47.102262 19472 leader_election.cc:288] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:47.104036 18141 leader_election.cc:302] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:47.111610 18027 leader_election.cc:302] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:47.112764 18293 raft_consensus.cc:2354] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3.
I20260807 03:38:47.114961 19488 consensus_queue.cc:1035] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 03:38:47.114733 17915 catalog_manager.cc:5495] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:38:47.116199 19497 raft_consensus.cc:482] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:47.121019 19497 raft_consensus.cc:3037] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:47.118507 19496 raft_consensus.cc:1050] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49: attempting to promote NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 to VOTER
I20260807 03:38:47.126237 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:47.127491 18190 raft_consensus.cc:2354] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 2. Current term is 3.
I20260807 03:38:47.128887 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:47.129904 18193 raft_consensus.cc:2454] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 2.
I20260807 03:38:47.133745 19475 raft_consensus.cc:2781] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:47.134459 19475 raft_consensus.cc:482] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:47.135140 19475 raft_consensus.cc:3037] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:47.146708 19499 raft_consensus.cc:2781] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 03:38:47.147686 19499 raft_consensus.cc:686] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:47.141399 19390 consensus_queue.cc:1035] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20260807 03:38:47.149159 19471 raft_consensus.cc:1050] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49: attempting to promote NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 to VOTER
I20260807 03:38:47.149390 19499 consensus_queue.cc:237] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:47.152923 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:47.153966 18309 raft_consensus.cc:2454] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:38:47.136118 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:47.170238 19295 raft_consensus.cc:1261] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 [term 1 LEARNER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 03:38:47.163549 18310 raft_consensus.cc:2454] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:38:47.173671 18248 leader_election.cc:398] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Vote denied by peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163) with higher term. Message: Invalid argument: T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 2. Current term is 3.
I20260807 03:38:47.181000 18249 leader_election.cc:398] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057) with higher term. Message: Invalid argument: T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 3. Current term is 4.
I20260807 03:38:47.181638 18141 leader_election.cc:398] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021) with higher term. Message: Invalid argument: T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3.
I20260807 03:38:47.183454 18295 raft_consensus.cc:3037] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 3
I20260807 03:38:47.183578 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:47.185896 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:47.186717 18180 raft_consensus.cc:3037] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 3
I20260807 03:38:47.187161 18187 raft_consensus.cc:2354] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 2. Current term is 3.
I20260807 03:38:47.188129 18248 leader_election.cc:401] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260807 03:38:47.188906 18249 leader_election.cc:401] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260807 03:38:47.189363 18141 leader_election.cc:401] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
W20260807 03:38:47.189653 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 44892.2us
I20260807 03:38:47.190773 18248 leader_election.cc:302] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:47.192019 19494 consensus_queue.cc:1035] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 03:38:47.197749 18311 raft_consensus.cc:1261] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:38:47.199374 18027 leader_election.cc:398] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Vote denied by peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021) with higher term. Message: Invalid argument: T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 2. Current term is 3.
I20260807 03:38:47.201946 18027 leader_election.cc:401] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260807 03:38:47.202726 18300 raft_consensus.cc:2454] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
W20260807 03:38:47.206319 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 68621.9us
I20260807 03:38:47.207329 18249 leader_election.cc:398] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057) with higher term. Message: Invalid argument: T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 2. Current term is 3.
I20260807 03:38:47.207687 18249 leader_election.cc:401] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260807 03:38:47.208984 19388 raft_consensus.cc:2726] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057) with higher term. Message: Invalid argument: T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 3. Current term is 4.
I20260807 03:38:47.209343 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:47.210178 19488 raft_consensus.cc:2781] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:47.210693 18184 raft_consensus.cc:2454] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 2.
I20260807 03:38:47.210965 19488 raft_consensus.cc:482] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:47.216167 19488 raft_consensus.cc:3037] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:47.212826 18248 leader_election.cc:302] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:47.213940 19390 raft_consensus.cc:2726] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057) with higher term. Message: Invalid argument: T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 2. Current term is 3.
I20260807 03:38:47.219774 18027 leader_election.cc:302] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:47.224985 18311 raft_consensus.cc:1261] T af8877c3fbba4ab18765832f1390921f P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 03:38:47.226855 18248 leader_election.cc:398] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Vote denied by peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163) with higher term. Message: Invalid argument: T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 2. Current term is 3.
I20260807 03:38:47.227341 18248 leader_election.cc:401] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260807 03:38:47.225756 18141 leader_election.cc:302] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:47.231495 19489 raft_consensus.cc:2781] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:38:47.232165 18027 leader_election.cc:302] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:47.233975 19344 raft_consensus.cc:2781] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:47.213909 19378 raft_consensus.cc:2726] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021) with higher term. Message: Invalid argument: T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 2. Current term is 3.
I20260807 03:38:47.234679 19344 raft_consensus.cc:482] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:47.236394 19489 raft_consensus.cc:686] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:38:47.248116 19344 raft_consensus.cc:3037] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:47.247098 18141 leader_election.cc:302] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:47.181473 19453 raft_consensus.cc:504] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:47.171324 19471 consensus_queue.cc:237] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:38:47.228448 19388 raft_consensus.cc:2781] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:38:47.282415 18248 leader_election.cc:302] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:47.282970 19473 raft_consensus.cc:2781] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:38:47.283957 19473 raft_consensus.cc:686] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:47.289738 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:38:47.290453 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 77031.8us
I20260807 03:38:47.228636 19469 consensus_queue.cc:1035] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.048s
I20260807 03:38:47.298130 18249 leader_election.cc:398] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057) with higher term. Message: Invalid argument: T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 2. Current term is 3.
I20260807 03:38:47.298414 18249 leader_election.cc:401] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260807 03:38:47.170184 19496 consensus_queue.cc:237] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:38:47.273396 19456 raft_consensus.cc:2781] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:38:47.277072 19478 leader_election.cc:288] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:47.301118 19472 raft_consensus.cc:2781] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:47.274168 19502 consensus_queue.cc:1035] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.084s
I20260807 03:38:47.332914 19473 consensus_queue.cc:237] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:47.319334 19472 raft_consensus.cc:482] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:47.337162 19472 raft_consensus.cc:3037] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:47.337987 19453 leader_election.cc:288] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:47.322708 19347 raft_consensus.cc:2726] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057) with higher term. Message: Invalid argument: T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 2. Current term is 3.
I20260807 03:38:47.315833 19456 raft_consensus.cc:686] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 3 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:47.367302 18295 raft_consensus.cc:1261] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:38:47.370451 18297 raft_consensus.cc:1261] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:38:47.386102 19468 raft_consensus.cc:2726] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163) with higher term. Message: Invalid argument: T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 2. Current term is 3.
I20260807 03:38:47.386397 19495 raft_consensus.cc:2726] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163) with higher term. Message: Invalid argument: T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 2. Current term is 3.
I20260807 03:38:47.361786 19489 consensus_queue.cc:237] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:47.399752 18180 raft_consensus.cc:2454] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3.
I20260807 03:38:47.400797 19492 raft_consensus.cc:2726] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021) with higher term. Message: Invalid argument: T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3.
I20260807 03:38:47.367441 19335 raft_consensus.cc:504] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:47.403987 18185 raft_consensus.cc:2354] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 2. Current term is 3.
I20260807 03:38:47.407998 19456 consensus_queue.cc:237] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:47.406687 19485 raft_consensus.cc:504] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:47.402702 19497 raft_consensus.cc:504] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:47.443567 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 158883us
I20260807 03:38:47.406762 19335 leader_election.cc:288] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:47.442646 19388 raft_consensus.cc:686] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:47.485424 18311 raft_consensus.cc:2454] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 2.
I20260807 03:38:47.504432 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:47.451793 19344 raft_consensus.cc:504] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:47.537914 18079 raft_consensus.cc:2354] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 2. Current term is 3.
W20260807 03:38:47.545627 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 272004us
I20260807 03:38:47.545879 19485 leader_election.cc:288] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
W20260807 03:38:47.562832 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 253068us
I20260807 03:38:47.569252 19488 raft_consensus.cc:504] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:47.556807 19388 consensus_queue.cc:237] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:47.571477 19497 leader_election.cc:288] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
W20260807 03:38:47.580153 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 333192us
I20260807 03:38:47.604761 19335 consensus_queue.cc:1035] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 03:38:47.606063 19499 consensus_queue.cc:1035] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20260807 03:38:47.622779 19295 raft_consensus.cc:1261] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 LEARNER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 03:38:47.624781 19485 consensus_queue.cc:1035] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
I20260807 03:38:47.641623 18027 leader_election.cc:302] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
W20260807 03:38:47.646361 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 95872.3us
I20260807 03:38:47.624069 19475 raft_consensus.cc:504] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:47.639812 19344 leader_election.cc:288] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:47.642225 19488 leader_election.cc:288] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:47.675369 18188 raft_consensus.cc:1261] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:38:47.653787 19492 consensus_queue.cc:1035] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20260807 03:38:47.682193 18027 leader_election.cc:398] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Vote denied by peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057) with higher term. Message: Invalid argument: T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 2. Current term is 3.
I20260807 03:38:47.683763 18027 leader_election.cc:401] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260807 03:38:47.692488 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:47.701813 19514 raft_consensus.cc:482] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:47.702430 19514 raft_consensus.cc:504] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:47.678149 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:47.695930 19378 raft_consensus.cc:2726] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057) with higher term. Message: Invalid argument: T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 2. Current term is 3.
I20260807 03:38:47.740306 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:47.742177 18309 raft_consensus.cc:3037] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:47.722388 19524 raft_consensus.cc:2781] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:47.756847 19516 raft_consensus.cc:482] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:47.724726 19472 raft_consensus.cc:504] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:47.745599 19524 raft_consensus.cc:482] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:47.772333 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:47.791592 18027 leader_election.cc:398] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Vote denied by peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057) with higher term. Message: Invalid argument: T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 2. Current term is 3.
I20260807 03:38:47.771461 19524 raft_consensus.cc:3037] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:47.780570 19514 leader_election.cc:288] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:47.819326 18310 raft_consensus.cc:2379] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 3: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:38:47.790792 19475 leader_election.cc:288] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:47.837410 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:47.838408 18311 raft_consensus.cc:3037] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 3
I20260807 03:38:47.833069 18184 raft_consensus.cc:1261] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 03:38:47.864874 19479 raft_consensus.cc:482] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:47.825366 19468 consensus_queue.cc:1035] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
W20260807 03:38:47.889881 19527 log.cc:924] Time spent T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.115s user 0.000s sys 0.002s
I20260807 03:38:47.891193 18189 raft_consensus.cc:2454] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
I20260807 03:38:47.783699 19516 raft_consensus.cc:504] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:47.857103 19511 raft_consensus.cc:482] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:47.866034 19479 raft_consensus.cc:504] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:47.900239 19472 leader_election.cc:288] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:47.925997 18309 raft_consensus.cc:2454] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3.
I20260807 03:38:47.918481 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:48.061133 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:47.917095 19511 raft_consensus.cc:504] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:48.073814 18185 raft_consensus.cc:2454] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
I20260807 03:38:48.086349 19483 raft_consensus.cc:482] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:48.095062 19483 raft_consensus.cc:504] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:48.089535 18311 raft_consensus.cc:2454] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3.
W20260807 03:38:48.136713 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 61789.7us
I20260807 03:38:48.145884 19516 leader_election.cc:288] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:38:48.152822 19534 log.cc:924] Time spent T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.069s user 0.001s sys 0.000s
I20260807 03:38:48.063273 19453 raft_consensus.cc:482] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:48.135138 19456 raft_consensus.cc:482] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:48.202471 19456 raft_consensus.cc:504] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:48.204790 19295 raft_consensus.cc:1261] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 1 LEARNER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 03:38:48.205081 19456 leader_election.cc:288] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:38:48.183027 19535 log.cc:924] Time spent T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.127s user 0.000s sys 0.001s
W20260807 03:38:48.195873 19536 log.cc:924] Time spent T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.090s user 0.000s sys 0.001s
I20260807 03:38:48.210567 19495 consensus_queue.cc:1035] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 03:38:48.251931 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:48.252800 19511 leader_election.cc:288] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:48.254681 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
W20260807 03:38:48.258667 19541 log.cc:924] Time spent T af8877c3fbba4ab18765832f1390921f P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
I20260807 03:38:48.263197 18295 raft_consensus.cc:1226] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260807 03:38:48.258810 18082 raft_consensus.cc:2354] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 2. Current term is 3.
I20260807 03:38:48.275164 19392 raft_consensus.cc:2932] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:38:48.282771 19511 raft_consensus.cc:482] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:48.283450 19511 raft_consensus.cc:504] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:48.288121 19511 leader_election.cc:288] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:48.297343 18300 raft_consensus.cc:1261] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:38:48.299201 19473 consensus_queue.cc:1035] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260807 03:38:48.300946 18140 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.558s user 0.091s sys 0.287s
W20260807 03:38:48.301270 18140 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.558s user 0.091s sys 0.287s
I20260807 03:38:48.302299 18248 leader_election.cc:398] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Vote denied by peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163) with higher term. Message: Invalid argument: T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 2. Current term is 3.
I20260807 03:38:48.302649 18248 leader_election.cc:401] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260807 03:38:48.257807 19516 raft_consensus.cc:482] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:48.246042 19483 leader_election.cc:288] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:48.318771 18248 leader_election.cc:398] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 election: Vote denied by peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163) with higher term. Message: Invalid argument: T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 2. Current term is 3.
W20260807 03:38:48.321534 18249 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.501s user 0.042s sys 0.050s
W20260807 03:38:48.324180 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 503548us
I20260807 03:38:48.323604 18081 raft_consensus.cc:1261] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:38:48.323089 18248 leader_election.cc:401] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260807 03:38:48.326200 19544 raft_consensus.cc:482] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:48.326872 19544 raft_consensus.cc:504] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:38:48.313053 18293 raft_consensus.cc:2932] T af8877c3fbba4ab18765832f1390921f P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:38:48.346427 19522 consensus_queue.cc:1046] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20260807 03:38:48.347329 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:48.347723 18075 raft_consensus.cc:1261] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:38:48.328572 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:48.314373 19344 raft_consensus.cc:482] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:48.361702 18300 raft_consensus.cc:1261] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:38:48.349256 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:48.246855 19453 raft_consensus.cc:504] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:48.368892 18027 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.574s user 0.016s sys 0.102s
W20260807 03:38:48.376389 18027 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.581s user 0.017s sys 0.102s
W20260807 03:38:48.376783 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 583467us
I20260807 03:38:48.383064 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:48.385169 18082 raft_consensus.cc:2454] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:38:48.358630 19479 leader_election.cc:288] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:48.388464 17915 catalog_manager.cc:5495] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:38:48.359112 19483 raft_consensus.cc:2726] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163) with higher term. Message: Invalid argument: T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 2. Current term is 3.
I20260807 03:38:48.389336 19490 raft_consensus.cc:2726] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163) with higher term. Message: Invalid argument: T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 2. Current term is 3.
I20260807 03:38:48.364162 18180 raft_consensus.cc:3037] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:48.319068 19516 raft_consensus.cc:504] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:48.409704 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:48.410625 18187 raft_consensus.cc:2379] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 3: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:38:48.363632 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:48.422616 19472 raft_consensus.cc:2932] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:38:48.420722 19512 raft_consensus.cc:482] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:48.429108 18193 raft_consensus.cc:1261] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 03:38:48.433202 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:48.435578 18081 raft_consensus.cc:3037] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:48.383222 19344 raft_consensus.cc:504] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:48.501668 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:48.430748 18071 raft_consensus.cc:3037] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:48.419492 19335 consensus_queue.cc:1035] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.090s
I20260807 03:38:48.536195 18193 raft_consensus.cc:1261] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:38:48.538546 18075 raft_consensus.cc:1261] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260807 03:38:48.531173 18138 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.562s user 0.020s sys 0.028s
I20260807 03:38:48.511307 19479 raft_consensus.cc:482] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:48.573148 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:48.502625 19546 raft_consensus.cc:482] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:48.576071 18190 raft_consensus.cc:3037] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
W20260807 03:38:48.574307 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 196563us
W20260807 03:38:48.574148 18138 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.828s user 0.021s sys 0.029s
I20260807 03:38:48.581338 18027 leader_election.cc:398] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Vote denied by peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057) with higher term. Message: Invalid argument: T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 2. Current term is 3.
I20260807 03:38:48.437512 19548 consensus_queue.cc:1035] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.145s
I20260807 03:38:48.596249 19559 raft_consensus.cc:482] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:48.596807 19559 raft_consensus.cc:504] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:48.598798 19559 leader_election.cc:288] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:48.514428 19512 raft_consensus.cc:504] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:38:48.602484 18075 raft_consensus.cc:1261] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 03:38:48.618038 18190 raft_consensus.cc:2454] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3.
I20260807 03:38:48.601013 19512 leader_election.cc:288] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:48.580827 19479 raft_consensus.cc:504] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:48.581099 19546 raft_consensus.cc:504] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:48.636241 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:48.640123 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:48.641449 18070 raft_consensus.cc:1261] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260807 03:38:48.661113 18180 raft_consensus.cc:2454] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3.
I20260807 03:38:48.661689 19557 raft_consensus.cc:482] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:48.634047 19453 leader_election.cc:288] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:48.696821 18082 raft_consensus.cc:3037] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
W20260807 03:38:48.709864 19554 log.cc:924] Time spent T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20260807 03:38:48.712500 17920 catalog_manager.cc:5495] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:48.712729 18190 raft_consensus.cc:2932] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
W20260807 03:38:48.717857 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 133373us
I20260807 03:38:48.721800 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:48.716197 19557 raft_consensus.cc:504] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:48.728386 19453 raft_consensus.cc:482] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:48.728840 19453 raft_consensus.cc:504] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:38:48.729667 19557 leader_election.cc:288] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:48.730135 19557 raft_consensus.cc:482] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:48.730564 19453 leader_election.cc:288] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:48.730521 19557 raft_consensus.cc:504] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:48.734155 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:48.735769 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:48.738229 19557 leader_election.cc:288] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:48.739939 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:48.736442 18073 raft_consensus.cc:2454] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:48.739567 18081 raft_consensus.cc:2454] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:38:48.744261 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:48.734478 19559 raft_consensus.cc:482] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:48.750931 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:48.757105 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:48.760828 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:48.741907 19344 leader_election.cc:288] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:48.761297 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:48.767923 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:48.756556 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:48.770907 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:48.753424 19392 raft_consensus.cc:482] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:48.769059 19471 raft_consensus.cc:482] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:48.779783 18298 raft_consensus.cc:1261] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:38:48.752828 19488 consensus_queue.cc:1035] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20260807 03:38:48.784416 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:48.785440 18180 raft_consensus.cc:2354] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 2. Current term is 3.
I20260807 03:38:48.784448 19545 raft_consensus.cc:482] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:48.788829 19545 raft_consensus.cc:504] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:48.745386 19523 consensus_queue.cc:1035] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
I20260807 03:38:48.806851 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:48.780158 19471 raft_consensus.cc:504] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:48.775019 19344 raft_consensus.cc:482] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:48.771610 19580 raft_consensus.cc:482] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:48.757200 19559 raft_consensus.cc:504] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:48.775445 18071 raft_consensus.cc:2454] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:38:48.776939 19571 raft_consensus.cc:482] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:48.813668 19579 raft_consensus.cc:482] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:48.809553 19580 raft_consensus.cc:504] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:48.792984 19546 leader_election.cc:288] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:48.792649 19479 leader_election.cc:288] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:48.813724 19571 raft_consensus.cc:504] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:38:48.816645 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:48.850081 18310 raft_consensus.cc:2354] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 2. Current term is 3.
I20260807 03:38:48.814972 19579 raft_consensus.cc:504] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:48.862995 19473 raft_consensus.cc:3032] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 03:38:48.864073 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:38:48.865249 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 126393us
I20260807 03:38:48.806372 19392 raft_consensus.cc:504] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:48.873098 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 114840us
I20260807 03:38:48.863461 19473 raft_consensus.cc:729] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:48.898070 18190 raft_consensus.cc:1261] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:38:48.784981 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:48.905616 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:48.906244 18193 raft_consensus.cc:2454] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:48.906620 19545 leader_election.cc:288] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:48.809124 19344 raft_consensus.cc:504] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:48.869901 19512 raft_consensus.cc:482] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:48.901151 19471 leader_election.cc:288] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:48.933949 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:49.062242 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:48.943970 18249 leader_election.cc:302] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
W20260807 03:38:49.094095 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 150224us
I20260807 03:38:49.061965 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:38:48.941672 19524 consensus_meta.cc:287] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 1.120s user 0.005s sys 0.028s
W20260807 03:38:48.989302 18183 consensus_meta.cc:287] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 0.926s user 0.010s sys 0.010s
I20260807 03:38:48.920364 19545 raft_consensus.cc:482] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:49.105490 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:49.112242 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:49.009459 19511 raft_consensus.cc:482] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:48.968245 19392 leader_election.cc:288] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:49.055622 19475 raft_consensus.cc:482] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:48.969892 19580 leader_election.cc:288] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:48.969101 19516 leader_election.cc:288] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:49.144505 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:48.971458 19579 leader_election.cc:288] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:49.146075 18079 raft_consensus.cc:3037] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:49.114571 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:48.970475 19344 leader_election.cc:288] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:49.015580 19576 raft_consensus.cc:482] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:49.093998 18082 raft_consensus.cc:2454] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:38:49.008177 19581 raft_consensus.cc:482] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:49.099187 19483 raft_consensus.cc:2781] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:49.160430 18073 raft_consensus.cc:1261] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 03:38:49.146940 19545 raft_consensus.cc:504] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:48.929116 19512 raft_consensus.cc:504] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:49.163326 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:49.165864 18070 raft_consensus.cc:1261] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:38:49.175427 19475 raft_consensus.cc:504] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:49.114229 18183 raft_consensus.cc:2454] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
I20260807 03:38:49.145732 19505 raft_consensus.cc:482] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:49.177079 19545 leader_election.cc:288] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:49.114372 19524 raft_consensus.cc:504] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:49.228004 19555 log.cc:924] Time spent T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331: Append to log took a long time: real 0.509s user 0.009s sys 0.004s
I20260807 03:38:49.166970 19483 raft_consensus.cc:482] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:49.232033 18300 raft_consensus.cc:1226] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260807 03:38:48.962126 19473 consensus_queue.cc:260] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:49.231146 19554 log.cc:924] Time spent T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
I20260807 03:38:49.237855 19546 raft_consensus.cc:482] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:49.248930 18069 raft_consensus.cc:2354] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 2. Current term is 3.
I20260807 03:38:49.240222 19505 raft_consensus.cc:504] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:49.247251 19480 raft_consensus.cc:482] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:49.240319 18183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44660 (request call id 665) took 1577 ms. Trace:
I20260807 03:38:49.257694 19524 leader_election.cc:288] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:49.257189 18183 rpcz_store.cc:276] 0807 03:38:47.657076 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:47.676719 (+ 19643us) service_pool.cc:227] Handling call
0807 03:38:48.060518 (+383799us) spinlock_profiling.cc:243] Waited 5.32 ms on lock 0x7b5800003998. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f25f851 00007fe98f25f702 00007fe99814fb09 00007fe99708b730 00007fe99715179b 00007fe997151821 00007fe9971011b0 00007fe997100a94 00007fe9907e8c04 00007fe9907e8b92
0807 03:38:49.234665 (+1174147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12237696}
I20260807 03:38:49.247402 19473 raft_consensus.cc:3037] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:49.164142 19511 raft_consensus.cc:504] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:49.254519 19505 leader_election.cc:288] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:49.240444 19483 raft_consensus.cc:3037] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:49.177539 19581 raft_consensus.cc:504] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:38:49.267048 19544 leader_election.cc:288] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:38:49.284018 18140 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.15.198.129:36163 timed out after 1.548s (SENT) [suppressed 4 similar messages]
I20260807 03:38:49.268793 19480 raft_consensus.cc:504] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:49.181210 19576 raft_consensus.cc:504] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:49.258780 19546 raft_consensus.cc:504] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:49.294605 19551 raft_consensus.cc:482] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:49.293413 19505 raft_consensus.cc:482] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:49.297087 19511 leader_election.cc:288] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:38:49.300822 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 114804us
I20260807 03:38:49.270143 19571 leader_election.cc:288] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:49.289158 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:49.280758 19581 leader_election.cc:288] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:38:49.292824 19480 leader_election.cc:288] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:38:49.336225 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 52243.2us
I20260807 03:38:49.332989 18079 raft_consensus.cc:2454] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:38:49.377481 19544 raft_consensus.cc:482] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:49.378325 19544 raft_consensus.cc:504] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:49.429428 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 337027us
I20260807 03:38:49.430687 18027 leader_election.cc:302] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:49.431728 18027 leader_election.cc:302] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:49.321010 19571 raft_consensus.cc:482] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
W20260807 03:38:49.438009 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 87009.7us
I20260807 03:38:49.311558 19505 raft_consensus.cc:504] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:49.312275 19551 raft_consensus.cc:504] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:49.387530 19475 leader_election.cc:288] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:49.440503 19581 raft_consensus.cc:2781] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:38:49.449064 19453 raft_consensus.cc:2781] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:38:49.436939 19499 raft_consensus.cc:482] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:49.483865 19581 raft_consensus.cc:686] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:38:49.448279 19524 raft_consensus.cc:482] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:49.480559 19392 consensus_queue.cc:1035] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20260807 03:38:49.501259 18027 leader_election.cc:398] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Vote denied by peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021) with higher term. Message: Invalid argument: T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 2. Current term is 3.
I20260807 03:38:49.542469 18027 leader_election.cc:401] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260807 03:38:49.442067 19571 raft_consensus.cc:504] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
W20260807 03:38:49.503010 19569 log.cc:924] Time spent T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.793s user 0.014s sys 0.016s
W20260807 03:38:49.502818 19587 log.cc:924] Time spent T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.122s user 0.000s sys 0.000s
I20260807 03:38:49.497025 18082 raft_consensus.cc:1261] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260807 03:38:49.502068 19591 log.cc:924] Time spent T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.189s user 0.000s sys 0.001s
I20260807 03:38:49.498780 19544 leader_election.cc:288] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:49.567747 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:49.500036 19453 raft_consensus.cc:686] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:38:49.506533 19576 leader_election.cc:288] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:49.509455 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:49.567256 18027 leader_election.cc:302] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:49.298580 19292 raft_consensus.cc:2932] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:38:49.500002 19522 consensus_queue.cc:1035] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.071s
I20260807 03:38:49.498879 19499 raft_consensus.cc:504] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:49.504175 19511 consensus_queue.cc:1035] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.066s
I20260807 03:38:49.580484 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:49.575881 19456 consensus_queue.cc:1035] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260807 03:38:49.581559 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:49.591640 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:49.592321 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:49.585786 18027 leader_election.cc:302] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:49.595255 18027 leader_election.cc:302] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:49.596596 18027 leader_election.cc:398] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Vote denied by peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057) with higher term. Message: Invalid argument: T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 2. Current term is 3.
I20260807 03:38:49.597223 18027 leader_election.cc:302] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:49.498610 19524 raft_consensus.cc:504] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:49.580365 19594 raft_consensus.cc:2726] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021) with higher term. Message: Invalid argument: T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 2. Current term is 3.
I20260807 03:38:49.502341 19581 consensus_queue.cc:237] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:49.600315 19479 raft_consensus.cc:2726] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 03:38:49.601143 19564 log.cc:924] Time spent T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.894s user 0.023s sys 0.029s
I20260807 03:38:49.587435 19576 raft_consensus.cc:2726] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:49.590184 19571 leader_election.cc:288] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:49.625190 18027 leader_election.cc:302] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
W20260807 03:38:49.598169 19593 log.cc:924] Time spent T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.109s user 0.000s sys 0.000s
I20260807 03:38:49.584693 19546 leader_election.cc:288] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:49.604727 19479 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:49.605877 19499 leader_election.cc:288] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:49.606768 19559 leader_election.cc:288] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:49.606683 18075 raft_consensus.cc:2932] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:38:49.598875 19475 raft_consensus.cc:2726] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:49.610036 19578 raft_consensus.cc:2726] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:49.610980 19551 leader_election.cc:288] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:49.611955 18081 raft_consensus.cc:1261] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:38:49.614670 19505 leader_election.cc:288] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:49.607884 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:49.661192 18079 raft_consensus.cc:2379] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 3: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:38:49.618194 19512 leader_election.cc:288] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:49.611966 18080 raft_consensus.cc:1261] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:38:49.593281 18082 raft_consensus.cc:1261] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260807 03:38:49.631896 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 61264.4us
I20260807 03:38:49.666226 18140 leader_election.cc:302] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:49.653126 19589 raft_consensus.cc:482] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
W20260807 03:38:49.656416 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 529299us
I20260807 03:38:49.659720 19515 raft_consensus.cc:482] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:49.639194 19594 raft_consensus.cc:2726] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:49.639453 19471 raft_consensus.cc:482] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:49.668474 18188 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44660 (request call id 676) took 1256 ms. Trace:
I20260807 03:38:49.651110 19388 raft_consensus.cc:482] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
W20260807 03:38:49.670809 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 44189.2us
I20260807 03:38:49.673596 18140 leader_election.cc:302] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:49.674086 19551 raft_consensus.cc:2726] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:49.682998 19524 leader_election.cc:288] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:49.671530 18188 rpcz_store.cc:276] 0807 03:38:48.409011 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:48.409127 (+ 116us) service_pool.cc:227] Handling call
0807 03:38:48.409445 (+ 318us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:38:48.424981 (+ 15536us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:38:48.425008 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:38:48.425437 (+ 429us) log.cc:842] Serialized 26 byte log entry
0807 03:38:48.702112 (+276675us) raft_consensus.cc:1631] Marking committed up to 1
0807 03:38:48.702164 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:38:48.702250 (+ 86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:38:49.502423 (+800173us) spinlock_profiling.cc:243] Waited 2.48 ms on lock 0x7b44000232cc. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f1a9a95 00007fe98f1aa4b2 00007fe991e52511 00007fe991e4de1a 00007fe991e41bee 00007fe991e40f36 00007fe991e40b58 00007fe991e4a551 00007fe991e4a509 00007fe991e4a499
0807 03:38:49.657208 (+154785us) raft_consensus.cc:1671] finished
0807 03:38:49.657230 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:38:49.664866 (+ 7636us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15143,"raft.queue_time_us":20048,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":7713,"spinlock_wait_cycles":5712256,"thread_start_us":276291,"threads_started":1,"wal-append.queue_time_us":284121}
I20260807 03:38:49.684957 18027 leader_election.cc:302] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:49.688691 18140 leader_election.cc:302] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:49.670117 18069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33254 (request call id 618) took 1011 ms. Trace:
I20260807 03:38:49.693219 18027 leader_election.cc:302] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:49.695993 18027 leader_election.cc:302] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:49.661715 19479 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:49.697950 18140 leader_election.cc:302] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:49.680408 19488 raft_consensus.cc:482] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:49.699913 19489 raft_consensus.cc:2726] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:49.704749 18140 leader_election.cc:302] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:49.677126 19580 consensus_queue.cc:1035] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20260807 03:38:49.696524 19578 raft_consensus.cc:2726] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:49.696924 19545 raft_consensus.cc:2781] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:38:49.705389 19493 raft_consensus.cc:2726] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 03:38:49.680341 19471 raft_consensus.cc:504] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:49.680982 19388 raft_consensus.cc:504] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:49.701359 19581 raft_consensus.cc:2726] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:49.724525 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:49.725346 19471 leader_election.cc:288] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:49.725937 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:49.717937 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:49.676716 19453 consensus_queue.cc:237] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:49.694303 19473 consensus_queue.cc:1035] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.013s
I20260807 03:38:49.685564 19515 raft_consensus.cc:504] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:49.692317 18069 rpcz_store.cc:276] 0807 03:38:48.659146 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:48.742494 (+ 83348us) service_pool.cc:227] Handling call
0807 03:38:49.669920 (+927426us) spinlock_profiling.cc:243] Waited 337 ms on lock 0x7b54002c42d4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:38:49.670061 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":775893376}
I20260807 03:38:49.703143 19488 raft_consensus.cc:504] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:49.719512 19551 raft_consensus.cc:2726] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:49.707003 19585 consensus_queue.cc:1035] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20260807 03:38:49.740447 19479 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:49.744333 19610 raft_consensus.cc:482] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:49.704267 19483 raft_consensus.cc:504] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:49.681491 19589 raft_consensus.cc:504] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:49.745038 19610 raft_consensus.cc:504] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:49.728350 19545 raft_consensus.cc:686] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:49.729321 19471 raft_consensus.cc:482] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:49.736304 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:49.727344 19522 consensus_queue.cc:1035] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20260807 03:38:49.772257 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:49.783974 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:49.795656 19571 raft_consensus.cc:482] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:49.795823 17912 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35098 (request call id 17) took 1092 ms. Trace:
I20260807 03:38:49.751470 19612 raft_consensus.cc:482] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
W20260807 03:38:49.780265 19568 log.cc:924] Time spent T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 1.058s user 0.011s sys 0.016s
I20260807 03:38:49.796216 17912 rpcz_store.cc:276] 0807 03:38:48.698092 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:48.706674 (+ 8582us) service_pool.cc:227] Handling call
0807 03:38:49.780548 (+1073874us) spinlock_profiling.cc:243] Waited 32.2 ms on lock 0x7b54002611d4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144af0 00007fe9921c7f2a 00007fe998119d6c 00007fe998213c3e 00007fe996525fb4 00007fe996525f42 00007fe996525e71 00007fe996525dec 00007fe9965250a2 00007fe9905e994c 00007fe9905e8d86
0807 03:38:49.781656 (+ 1108us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":0,"spinlock_wait_cycles":74030336}
I20260807 03:38:49.764398 19471 raft_consensus.cc:504] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:49.817108 19515 leader_election.cc:288] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:49.806857 19612 raft_consensus.cc:504] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:49.800307 19571 raft_consensus.cc:504] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:38:49.873147 18295 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57268 (request call id 637) took 1550 ms. Trace:
I20260807 03:38:49.873682 18295 rpcz_store.cc:276] 0807 03:38:48.322489 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:48.323768 (+ 1279us) service_pool.cc:227] Handling call
0807 03:38:48.347531 (+ 23763us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:38:48.371894 (+ 24363us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:38:48.376505 (+ 4611us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:38:48.509555 (+133050us) log.cc:842] Serialized 26 byte log entry
0807 03:38:48.632337 (+122782us) raft_consensus.cc:1631] Marking committed up to 1
0807 03:38:48.634049 (+ 1712us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:38:48.649985 (+ 15936us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:38:49.703471 (+1053486us) spinlock_profiling.cc:243] Waited 2.55 ms on lock 0x7b44000232cc. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f1a9a95 00007fe98f1aa4b2 00007fe991e52511 00007fe991e4de1a 00007fe991e41bee 00007fe991e40f36 00007fe991e40b58 00007fe991e4a551 00007fe991e4a509 00007fe991e4a499
0807 03:38:49.872742 (+169271us) raft_consensus.cc:1671] finished
0807 03:38:49.872775 (+ 33us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:38:49.873042 (+ 267us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12490,"raft.queue_time_us":24128,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":16564,"spinlock_wait_cycles":5856000,"thread_start_us":118594,"threads_started":1,"wal-append.queue_time_us":200636}
I20260807 03:38:49.891479 19610 leader_election.cc:288] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:49.857782 19545 consensus_queue.cc:237] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:49.924782 18185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44660 (request call id 692) took 1026 ms. Trace:
I20260807 03:38:49.925372 18185 rpcz_store.cc:276] 0807 03:38:48.898109 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:48.903148 (+ 5039us) service_pool.cc:227] Handling call
0807 03:38:49.678486 (+775338us) spinlock_profiling.cc:243] Waited 27 ms on lock 0x7b5800003998. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f25f851 00007fe98f25f702 00007fe99814fb09 00007fe99708b730 00007fe99715179b 00007fe997151821 00007fe9971152b0 00007fe9971007f9 00007fe9971002a2 00007fe9907e6684
0807 03:38:49.923728 (+245242us) spinlock_profiling.cc:243] Waited 1.62 ms on lock 0x7b5800003998. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f25fb7c 00007fe98f25f73b 00007fe99814fb09 00007fe99708b730 00007fe99715179b 00007fe997151821 00007fe9971152b0 00007fe9971007f9 00007fe9971002a2 00007fe9907e6684
0807 03:38:49.923967 (+ 239us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:38:49.924219 (+ 252us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:38:49.924262 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:38:49.924439 (+ 177us) raft_consensus.cc:1631] Marking committed up to 1
0807 03:38:49.924490 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:38:49.924569 (+ 79us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:38:49.924704 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":65937920}
I20260807 03:38:49.927950 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:49.946669 18189 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44660 (request call id 694) took 1042 ms. Trace:
I20260807 03:38:49.948346 19544 raft_consensus.cc:2781] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:49.950677 19544 raft_consensus.cc:482] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:49.951184 19544 raft_consensus.cc:3037] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:49.974614 17915 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39510 (request call id 138) took 1686 ms. Trace:
I20260807 03:38:49.977159 17920 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39492 (request call id 143) took 1591 ms. Trace:
I20260807 03:38:49.977483 17920 rpcz_store.cc:276] 0807 03:38:48.386191 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:48.386362 (+ 171us) service_pool.cc:227] Handling call
0807 03:38:49.977102 (+1590740us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 03:38:48.774347 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 03:38:48.808289 (+ 33942us) write_op.cc:432] Acquiring schema lock in shared mode
0807 03:38:48.808323 (+ 34us) write_op.cc:435] Acquired schema lock
0807 03:38:48.808351 (+ 28us) tablet.cc:660] Decoding operations
0807 03:38:48.808654 (+ 303us) write_op.cc:620] Acquiring the partition lock for write op
0807 03:38:48.808693 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0807 03:38:48.808722 (+ 29us) tablet.cc:683] Acquiring locks for 1 operations
0807 03:38:48.808880 (+ 158us) tablet.cc:699] Row locks acquired
0807 03:38:48.808896 (+ 16us) write_op.cc:260] PREPARE: finished
0807 03:38:48.809017 (+ 121us) write_op.cc:270] Start()
0807 03:38:48.809128 (+ 111us) write_op.cc:276] Timestamp: P: 1786073928808980 usec, L: 0
0807 03:38:48.809150 (+ 22us) op_driver.cc:331] REPLICATION: starting
0807 03:38:48.809676 (+ 526us) log.cc:842] Serialized 543 byte log entry
0807 03:38:49.440141 (+630465us) op_driver.cc:444] REPLICATION: finished
0807 03:38:49.590984 (+150843us) write_op.cc:301] APPLY: starting
0807 03:38:49.591045 (+ 61us) tablet.cc:1365] starting BulkCheckPresence
0807 03:38:49.591133 (+ 88us) tablet.cc:1368] finished BulkCheckPresence
0807 03:38:49.591236 (+ 103us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 03:38:49.591442 (+ 206us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 03:38:49.591589 (+ 147us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0807 03:38:49.591639 (+ 50us) write_op.cc:312] APPLY: finished
0807 03:38:49.604034 (+ 12395us) log.cc:842] Serialized 22 byte log entry
0807 03:38:49.649022 (+ 44988us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:38:49.651853 (+ 2831us) write_op.cc:454] Released schema lock
0807 03:38:49.654752 (+ 2899us) write_op.cc:341] FINISH: Updating metrics
0807 03:38:49.670660 (+ 15908us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:38:49.672445 (+ 1785us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":314866,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":140263,"apply.run_cpu_time_us":1999,"apply.run_wall_time_us":90374,"num_ops":1,"prepare.queue_time_us":42413,"prepare.run_cpu_time_us":2290,"prepare.run_wall_time_us":83565,"replication_time_us":619745,"thread_start_us":139035,"threads_started":1}]]}
I20260807 03:38:50.092628 19589 leader_election.cc:288] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:49.981267 19571 leader_election.cc:288] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:50.089147 19488 leader_election.cc:288] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:50.089138 19483 leader_election.cc:288] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:50.098488 19548 raft_consensus.cc:482] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:50.065296 19583 raft_consensus.cc:482] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:50.151511 19471 leader_election.cc:288] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:50.096539 19515 raft_consensus.cc:482] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:50.034803 19480 raft_consensus.cc:482] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:50.146459 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:50.143445 19609 raft_consensus.cc:482] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:50.150882 19523 raft_consensus.cc:482] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:50.151680 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:50.152942 18249 leader_election.cc:302] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:49.975001 17915 rpcz_store.cc:276] 0807 03:38:48.288597 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:48.288835 (+ 238us) service_pool.cc:227] Handling call
0807 03:38:49.974536 (+1685701us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 03:38:48.596449 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 03:38:48.597016 (+ 567us) write_op.cc:432] Acquiring schema lock in shared mode
0807 03:38:48.597049 (+ 33us) write_op.cc:435] Acquired schema lock
0807 03:38:48.597067 (+ 18us) tablet.cc:660] Decoding operations
0807 03:38:48.597384 (+ 317us) write_op.cc:620] Acquiring the partition lock for write op
0807 03:38:48.597442 (+ 58us) write_op.cc:641] Partition lock acquired for write op
0807 03:38:48.597465 (+ 23us) tablet.cc:683] Acquiring locks for 1 operations
0807 03:38:48.597632 (+ 167us) tablet.cc:699] Row locks acquired
0807 03:38:48.597648 (+ 16us) write_op.cc:260] PREPARE: finished
0807 03:38:48.597777 (+ 129us) write_op.cc:270] Start()
0807 03:38:48.597907 (+ 130us) write_op.cc:276] Timestamp: P: 1786073928597737 usec, L: 0
0807 03:38:48.597923 (+ 16us) op_driver.cc:331] REPLICATION: starting
0807 03:38:48.598549 (+ 626us) log.cc:842] Serialized 543 byte log entry
0807 03:38:48.991299 (+392750us) op_driver.cc:444] REPLICATION: finished
0807 03:38:49.589039 (+597740us) spinlock_profiling.cc:243] Waited 5.87 ms on lock 0x7b18000eae80. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe9981368b1 00007fe998143661 00007fe98f297638 00007fe98f29ab11 00007fe98f29aac9 00007fe98f29aa59 00007fe98f29aa21 00007fe98f299d1d 00007fe9984fb204 00007fe9984fb039 00007fe98f279dc6
0807 03:38:49.589184 (+ 145us) write_op.cc:301] APPLY: starting
0807 03:38:49.589269 (+ 85us) tablet.cc:1365] starting BulkCheckPresence
0807 03:38:49.589363 (+ 94us) tablet.cc:1368] finished BulkCheckPresence
0807 03:38:49.589392 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 03:38:49.589635 (+ 243us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 03:38:49.589785 (+ 150us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0807 03:38:49.589828 (+ 43us) write_op.cc:312] APPLY: finished
0807 03:38:49.590023 (+ 195us) log.cc:842] Serialized 22 byte log entry
0807 03:38:49.590182 (+ 159us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:38:49.590255 (+ 73us) write_op.cc:454] Released schema lock
0807 03:38:49.590290 (+ 35us) write_op.cc:341] FINISH: Updating metrics
0807 03:38:49.590553 (+ 263us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:38:49.590590 (+ 37us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":309829,"apply.run_cpu_time_us":1615,"apply.run_wall_time_us":1616,"num_ops":1,"prepare.queue_time_us":204404,"prepare.run_cpu_time_us":22085,"prepare.run_wall_time_us":147403,"raft.queue_time_us":507538,"raft.run_cpu_time_us":125370,"raft.run_wall_time_us":660869,"replication_time_us":378141,"spinlock_wait_cycles":13510400,"thread_start_us":597827,"threads_started":4,"wal-append.queue_time_us":190157}]]}
I20260807 03:38:50.139250 19456 raft_consensus.cc:482] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:49.997831 19612 leader_election.cc:288] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:50.140059 19388 leader_election.cc:288] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:49.947248 18189 rpcz_store.cc:276] 0807 03:38:48.904622 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:48.906227 (+ 1605us) service_pool.cc:227] Handling call
0807 03:38:48.906716 (+ 489us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:38:48.907080 (+ 364us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:38:48.907105 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:38:48.940387 (+ 33282us) log.cc:842] Serialized 572 byte log entry
0807 03:38:49.945959 (+1005572us) spinlock_profiling.cc:243] Waited 35.7 ms on lock 0x7b54000797a0. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144af0 00007fe99214f028 00007fe9921bfa7b 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c
0807 03:38:49.946048 (+ 89us) raft_consensus.cc:1631] Marking committed up to 2
0807 03:38:49.946118 (+ 70us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:38:49.946216 (+ 98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:38:49.946280 (+ 64us) raft_consensus.cc:1671] finished
0807 03:38:49.946297 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:38:49.946577 (+ 280us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":82200320,"thread_start_us":155732,"threads_started":1,"wal-append.queue_time_us":149459}
I20260807 03:38:50.179822 19469 raft_consensus.cc:482] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:50.190438 17914 catalog_manager.cc:5495] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:50.226668 18300 raft_consensus.cc:1261] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
W20260807 03:38:50.228585 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 75719.4us
I20260807 03:38:50.223248 17914 catalog_manager.cc:5495] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:50.231500 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:50.204972 19610 raft_consensus.cc:482] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:50.243530 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:50.207468 19496 raft_consensus.cc:482] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:50.248370 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:50.251441 18190 raft_consensus.cc:2396] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 03:38:50.249306 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:50.249935 19551 raft_consensus.cc:482] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:50.179807 19548 raft_consensus.cc:504] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:50.250231 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:50.207448 18298 raft_consensus.cc:2454] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:50.195770 19515 raft_consensus.cc:504] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:50.241191 19388 raft_consensus.cc:2781] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:38:50.241250 19612 raft_consensus.cc:482] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:50.207492 19456 raft_consensus.cc:504] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:50.250531 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:50.190430 19583 raft_consensus.cc:504] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:50.205484 19480 raft_consensus.cc:504] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:50.296516 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:50.210176 19582 raft_consensus.cc:482] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:50.228752 19469 raft_consensus.cc:504] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:50.200289 19523 raft_consensus.cc:504] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:50.437250 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:50.277074 17914 catalog_manager.cc:5495] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
W20260807 03:38:50.438896 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 168668us
I20260807 03:38:50.201089 19609 raft_consensus.cc:504] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:50.441341 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:38:50.438256 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 195070us
I20260807 03:38:50.442386 19469 leader_election.cc:288] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:50.445825 18312 raft_consensus.cc:2454] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:50.453451 18188 raft_consensus.cc:2454] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:38:50.455804 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:50.456560 18305 raft_consensus.cc:2454] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:38:50.456637 19495 consensus_queue.cc:1035] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 03:38:50.461349 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:50.437497 19548 leader_election.cc:288] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:50.317085 18310 raft_consensus.cc:2454] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:50.408414 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:50.488302 18300 raft_consensus.cc:2454] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:50.489037 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:50.491981 18309 raft_consensus.cc:2454] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:50.465590 19582 raft_consensus.cc:504] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:50.481266 19627 log.cc:924] Time spent T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.244s user 0.000s sys 0.001s
I20260807 03:38:50.294869 19551 raft_consensus.cc:504] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:50.280397 19557 raft_consensus.cc:482] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:50.494971 19583 leader_election.cc:288] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:50.252449 19610 raft_consensus.cc:504] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:50.502002 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:50.493613 19471 consensus_queue.cc:1035] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260807 03:38:50.348423 18138 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.132:41381, user_credentials={real_user=slave}} blocked reactor thread for 53817.3us
I20260807 03:38:50.577082 18193 raft_consensus.cc:2454] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
W20260807 03:38:50.583427 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 89776.2us
I20260807 03:38:50.592698 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:50.596506 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:50.280128 19496 raft_consensus.cc:504] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:50.465575 17914 catalog_manager.cc:5495] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: config changed from index 2 to 3, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260807 03:38:50.600523 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 102659us
I20260807 03:38:50.600678 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:50.602815 18193 raft_consensus.cc:2454] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:38:50.492058 19612 raft_consensus.cc:504] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:50.606129 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:50.612066 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:50.613371 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:50.589041 19594 raft_consensus.cc:482] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:50.583241 19388 raft_consensus.cc:686] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 2 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:50.597438 19578 raft_consensus.cc:482] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:50.585157 19596 raft_consensus.cc:482] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:50.570623 19557 raft_consensus.cc:504] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:50.604429 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:50.635470 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:50.628479 19578 raft_consensus.cc:504] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:50.636818 19557 leader_election.cc:288] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:50.627827 19625 raft_consensus.cc:482] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:50.642539 19625 raft_consensus.cc:504] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:50.624797 19483 raft_consensus.cc:482] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:50.644924 19625 leader_election.cc:288] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:50.645294 19483 raft_consensus.cc:504] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:50.625181 19583 raft_consensus.cc:482] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
W20260807 03:38:50.639503 19637 log.cc:924] Time spent T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.145s user 0.000s sys 0.001s
I20260807 03:38:50.628962 19582 leader_election.cc:288] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:50.637965 19515 leader_election.cc:288] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:50.649959 18189 raft_consensus.cc:3037] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:50.632499 19594 raft_consensus.cc:504] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:50.638469 19619 consensus_queue.cc:1035] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.038s
I20260807 03:38:50.666927 19483 leader_election.cc:288] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:50.672502 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:50.681181 19456 leader_election.cc:288] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:50.683378 19480 leader_election.cc:288] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:50.665933 19583 raft_consensus.cc:504] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:50.681064 19523 leader_election.cc:288] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:50.692692 19297 raft_consensus.cc:2932] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:38:50.729022 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:50.730302 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:50.730410 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:50.731142 18187 raft_consensus.cc:2454] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:38:50.731952 19594 leader_election.cc:288] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:50.732967 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:50.733043 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:50.733904 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:50.734727 18072 raft_consensus.cc:2454] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 1.
I20260807 03:38:50.735216 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:50.735805 18070 raft_consensus.cc:2454] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 1.
I20260807 03:38:50.747263 18190 raft_consensus.cc:2396] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260807 03:38:50.719518 19453 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s user 0.017s sys 0.035s
W20260807 03:38:50.734673 19639 log.cc:924] Time spent T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.131s user 0.001s sys 0.001s
I20260807 03:38:50.753314 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:50.650394 19596 raft_consensus.cc:504] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:38:50.756294 18073 raft_consensus.cc:2454] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:50.747365 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:38:50.751284 19453 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.575s user 0.017s sys 0.035s
I20260807 03:38:50.759905 19578 leader_election.cc:288] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:50.761475 19604 raft_consensus.cc:482] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:50.761637 19239 leader_election.cc:302] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:50.763942 19604 raft_consensus.cc:504] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:50.766232 19604 leader_election.cc:288] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:50.718873 19551 leader_election.cc:288] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:50.767246 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:38:50.771291 19638 log.cc:924] Time spent T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.155s user 0.001s sys 0.000s
W20260807 03:38:50.773178 19634 log.cc:924] Time spent T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.135s user 0.000s sys 0.000s
I20260807 03:38:50.723073 19610 leader_election.cc:288] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:50.723116 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:50.720232 18298 raft_consensus.cc:2454] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:38:50.778908 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:50.692831 19609 leader_election.cc:288] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:50.754914 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:50.790577 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:50.804203 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:50.787084 19496 leader_election.cc:288] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:50.722419 19388 consensus_queue.cc:237] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:50.789402 19612 leader_election.cc:288] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:50.811669 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:50.814549 18293 raft_consensus.cc:2379] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 3: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:38:50.818876 19583 leader_election.cc:288] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:50.820439 19596 leader_election.cc:288] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:38:50.824915 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 204941us
I20260807 03:38:50.826083 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:50.827893 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:38:50.830120 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 203542us
I20260807 03:38:50.831722 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:50.814651 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:50.833088 18248 leader_election.cc:302] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:50.838199 18248 leader_election.cc:398] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 election: Vote denied by peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163) with higher term. Message: Invalid argument: T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 2. Current term is 3.
I20260807 03:38:50.838976 18248 leader_election.cc:302] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:50.839279 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:50.842814 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:50.840554 19609 raft_consensus.cc:2726] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 03:38:50.844003 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:50.848701 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:50.849956 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:50.851025 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:50.841403 19523 raft_consensus.cc:2726] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:50.837642 19615 consensus_queue.cc:1035] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20260807 03:38:50.869863 19240 leader_election.cc:302] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
W20260807 03:38:50.841670 19239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 80124.1us
I20260807 03:38:50.874503 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:50.875548 18293 raft_consensus.cc:2396] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260807 03:38:50.884618 19544 consensus_meta.cc:287] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 0.933s user 0.014s sys 0.001s
I20260807 03:38:50.885167 19544 raft_consensus.cc:504] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:38:50.864070 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:50.852082 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:50.893545 19624 raft_consensus.cc:2932] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:38:50.905308 19240 leader_election.cc:302] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:50.908566 19544 leader_election.cc:288] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:38:50.905984 19292 raft_consensus.cc:2932] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:38:50.912834 19669 raft_consensus.cc:2726] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:50.925277 19669 raft_consensus.cc:2726] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:50.927080 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:50.927747 19295 raft_consensus.cc:2932] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:38:50.931980 19596 raft_consensus.cc:2726] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:50.955003 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:50.971464 18082 raft_consensus.cc:2454] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
W20260807 03:38:50.972952 19240 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 67756.6us
I20260807 03:38:50.978317 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:50.995055 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:50.989367 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:50.999630 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:51.000970 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:51.002187 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:51.003492 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:50.993185 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:51.005544 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:51.007223 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:50.984587 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:51.017987 18187 raft_consensus.cc:2454] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
I20260807 03:38:51.018658 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:51.019770 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:51.021160 18193 raft_consensus.cc:2379] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5c9b5062bdfa46af9396f1a85025a331 in current term 2: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:38:51.021914 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:51.022745 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:51.023504 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:51.023392 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:50.997709 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:51.032521 18183 raft_consensus.cc:2454] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:38:51.024583 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:51.024701 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:51.025803 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:51.033340 18190 raft_consensus.cc:2454] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:38:51.032650 19617 raft_consensus.cc:2932] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:38:51.026185 18188 raft_consensus.cc:2454] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:51.026739 19295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331"
I20260807 03:38:51.029269 19240 leader_election.cc:302] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:51.029670 18249 leader_election.cc:302] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:51.051789 18249 leader_election.cc:302] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:51.048563 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:51.054796 18180 raft_consensus.cc:2454] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:38:51.049372 19488 raft_consensus.cc:2781] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:51.056542 19488 raft_consensus.cc:482] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:51.057363 19488 raft_consensus.cc:3037] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:51.054296 19624 raft_consensus.cc:2781] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:38:51.057793 19624 raft_consensus.cc:482] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:51.058182 19624 raft_consensus.cc:3037] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:38:51.062449 18249 leader_election.cc:302] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:51.064352 19686 raft_consensus.cc:3037] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:51.068269 19496 raft_consensus.cc:2781] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:51.068648 19496 raft_consensus.cc:482] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:51.069015 19496 raft_consensus.cc:3037] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:51.070430 18249 leader_election.cc:302] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:51.071163 19495 raft_consensus.cc:2781] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:38:51.071553 19495 raft_consensus.cc:482] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:51.071890 19495 raft_consensus.cc:3037] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:38:51.079311 19496 raft_consensus.cc:504] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:51.080113 19645 raft_consensus.cc:482] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:51.083546 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:51.080793 19645 raft_consensus.cc:504] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:51.086196 18310 raft_consensus.cc:1226] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:51.082996 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:51.097648 19624 raft_consensus.cc:504] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:51.100234 19495 raft_consensus.cc:504] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:51.093283 19578 raft_consensus.cc:482] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:51.102818 19578 raft_consensus.cc:504] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:51.107523 19686 raft_consensus.cc:2726] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:51.108295 19594 raft_consensus.cc:482] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:51.108913 19594 raft_consensus.cc:504] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:51.109690 19496 leader_election.cc:288] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:51.111781 19495 leader_election.cc:288] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:51.113715 19578 leader_election.cc:288] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:51.100936 19488 raft_consensus.cc:504] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:51.116537 19544 raft_consensus.cc:482] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:51.101207 19650 raft_consensus.cc:482] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:51.117038 19544 raft_consensus.cc:504] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:51.117300 19650 raft_consensus.cc:504] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:51.117908 19488 leader_election.cc:288] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:51.118978 19544 leader_election.cc:288] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:51.119146 19650 leader_election.cc:288] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:51.119315 18311 raft_consensus.cc:1226] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 2. Current term is 3. Ops: []
I20260807 03:38:51.091918 19645 leader_election.cc:288] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:51.113807 19594 leader_election.cc:288] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:51.121294 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:51.122557 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:51.123636 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:51.101070 19624 leader_election.cc:288] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:51.060791 18180 raft_consensus.cc:2932] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:38:51.115223 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:51.127676 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:51.128859 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:51.129179 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:51.130404 18297 raft_consensus.cc:1261] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:38:51.131901 19583 consensus_queue.cc:1035] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:51.132330 18310 raft_consensus.cc:2379] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 4: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:38:51.133436 19594 raft_consensus.cc:482] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:51.134812 18027 leader_election.cc:302] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:38:51.136546 18141 leader_election.cc:302] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:51.137223 19551 raft_consensus.cc:2781] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:51.137580 19551 raft_consensus.cc:482] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:51.126114 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:51.138942 18141 leader_election.cc:302] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:51.143198 18141 leader_election.cc:302] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:51.145958 18141 leader_election.cc:302] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:51.137986 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:51.147054 19505 raft_consensus.cc:2781] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:51.148029 19505 raft_consensus.cc:482] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:51.148563 19505 raft_consensus.cc:3037] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:51.137945 19551 raft_consensus.cc:3037] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:51.134181 19594 raft_consensus.cc:504] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:38:51.138023 18189 raft_consensus.cc:2454] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3.
W20260807 03:38:51.166373 17914 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.509s user 0.002s sys 0.008s
I20260807 03:38:51.146206 19480 raft_consensus.cc:2781] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:51.163457 19488 raft_consensus.cc:482] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:51.165294 19594 leader_election.cc:288] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:51.166240 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:51.145612 19645 raft_consensus.cc:2781] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260807 03:38:51.167448 17914 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.513s user 0.004s sys 0.008s
I20260807 03:38:51.169363 18189 raft_consensus.cc:3037] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:51.169194 19645 raft_consensus.cc:482] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:51.168776 19480 raft_consensus.cc:482] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:51.169018 19488 raft_consensus.cc:504] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:51.174386 19645 raft_consensus.cc:3037] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:51.174520 19480 raft_consensus.cc:3037] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:51.199220 18141 leader_election.cc:302] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:51.205135 19701 raft_consensus.cc:482] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:51.205713 19701 raft_consensus.cc:504] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:51.203402 19488 leader_election.cc:288] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:51.218690 19551 raft_consensus.cc:504] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:51.220105 19650 consensus_queue.cc:1046] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:51.220709 19544 raft_consensus.cc:2781] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:51.221149 19544 raft_consensus.cc:482] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:51.221601 19544 raft_consensus.cc:3037] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:51.221221 19650 raft_consensus.cc:3032] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:51.223831 19650 raft_consensus.cc:729] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:38:51.220077 19701 leader_election.cc:288] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:51.229627 19551 leader_election.cc:288] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:51.232071 19505 raft_consensus.cc:504] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:51.232923 18080 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33268 (request call id 607) took 1516 ms. Trace:
I20260807 03:38:51.234196 18080 rpcz_store.cc:276] 0807 03:38:49.716880 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:49.933573 (+216693us) service_pool.cc:227] Handling call
0807 03:38:49.934004 (+ 431us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:38:50.098501 (+164497us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:38:50.098547 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:38:50.100946 (+ 2399us) log.cc:842] Serialized 26 byte log entry
0807 03:38:50.229745 (+128799us) raft_consensus.cc:1631] Marking committed up to 1
0807 03:38:50.229786 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:38:50.229852 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:38:50.252019 (+ 22167us) spinlock_profiling.cc:243] Waited 1.35 ms on lock 0x7b4800281958. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe9970667f1 00007fe99705d8e7 00007fe99705d77a 00007fe99715195f 00007fe997151627 00007fe997165b2c 00007fe997165ad8 00007fe997165a58 00007fe997165a13 00007fe997164cbc 00007fe9960e6be6
0807 03:38:51.232523 (+980504us) raft_consensus.cc:1671] finished
0807 03:38:51.232555 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:38:51.232828 (+ 273us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":164163,"raft.queue_time_us":183040,"raft.run_cpu_time_us":1570,"raft.run_wall_time_us":122969,"spinlock_wait_cycles":3095424,"thread_start_us":74615,"threads_started":1,"wal-append.queue_time_us":373328}
I20260807 03:38:51.241219 19505 leader_election.cc:288] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:51.249953 18027 leader_election.cc:302] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
W20260807 03:38:51.258582 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 59444.6us
I20260807 03:38:51.261031 18027 leader_election.cc:302] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:51.263324 18027 leader_election.cc:302] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:51.267306 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:51.264632 19641 raft_consensus.cc:2726] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:51.267966 18180 raft_consensus.cc:3037] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:38:51.270285 19479 raft_consensus.cc:2726] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:51.245970 19650 consensus_queue.cc:260] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:51.737490 19650 raft_consensus.cc:3037] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
W20260807 03:38:51.281575 19642 log.cc:924] Time spent T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.593s user 0.003s sys 0.000s
I20260807 03:38:51.739910 18180 raft_consensus.cc:2454] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
I20260807 03:38:51.741915 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:51.300609 18189 raft_consensus.cc:2454] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:38:51.743208 18180 raft_consensus.cc:3037] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:51.743722 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:51.744684 18189 raft_consensus.cc:3037] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:38:51.474305 19480 raft_consensus.cc:504] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:51.465519 19718 log.cc:924] Time spent T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.201s user 0.001s sys 0.000s
I20260807 03:38:51.731479 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:52.060564 19480 leader_election.cc:288] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:51.733413 18027 leader_election.cc:302] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
W20260807 03:38:51.735052 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 214756us
I20260807 03:38:52.061092 18070 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33268 (request call id 615) took 1323 ms. Trace:
W20260807 03:38:52.061834 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 328482us
I20260807 03:38:52.062033 18141 leader_election.cc:302] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:52.062178 19479 raft_consensus.cc:2726] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:52.062686 18027 leader_election.cc:302] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:52.061570 18070 rpcz_store.cc:276] 0807 03:38:50.737567 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:50.737965 (+ 398us) service_pool.cc:227] Handling call
0807 03:38:51.219243 (+481278us) spinlock_profiling.cc:243] Waited 6.06 ms on lock 0x7b5800001298. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f25f851 00007fe98f25f702 00007fe99814fb09 00007fe99708b730 00007fe99715179b 00007fe997151821 00007fe9971152b0 00007fe9971007f9 00007fe9971002a2 00007fe9907e6684
0807 03:38:51.219449 (+ 206us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:38:51.246594 (+ 27145us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:38:51.246638 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:38:51.249019 (+ 2381us) log.cc:842] Serialized 26 byte log entry
0807 03:38:51.263202 (+ 14183us) raft_consensus.cc:1631] Marking committed up to 2
0807 03:38:51.263532 (+ 330us) log.cc:842] Serialized 14 byte log entry
0807 03:38:51.267273 (+ 3741us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:38:51.267345 (+ 72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:38:52.060814 (+793469us) raft_consensus.cc:1671] finished
0807 03:38:52.060916 (+ 102us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:38:52.061034 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26691,"prepare.queue_time_us":172,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":27,"raft.queue_time_us":30245,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":672,"spinlock_wait_cycles":13942528,"thread_start_us":10484,"threads_started":1,"wal-append.queue_time_us":10979}
I20260807 03:38:52.063921 18027 leader_election.cc:302] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:52.064443 19492 consensus_queue.cc:1035] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:52.064590 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:51.255985 19705 consensus_queue.cc:1046] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.250s
I20260807 03:38:52.067162 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:38:51.730176 19645 consensus_meta.cc:287] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 0.555s user 0.020s sys 0.000s
I20260807 03:38:52.096426 19645 raft_consensus.cc:504] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:51.755362 18069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33268 (request call id 612) took 2036 ms. Trace:
I20260807 03:38:51.762791 18180 raft_consensus.cc:2454] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
I20260807 03:38:51.768518 18189 raft_consensus.cc:2454] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
I20260807 03:38:52.098505 18069 rpcz_store.cc:276] 0807 03:38:49.718333 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:49.732441 (+ 14108us) service_pool.cc:227] Handling call
0807 03:38:49.732908 (+ 467us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:38:50.139330 (+406422us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:38:50.144008 (+ 4678us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:38:50.151715 (+ 7707us) log.cc:842] Serialized 26 byte log entry
0807 03:38:50.586425 (+434710us) raft_consensus.cc:1631] Marking committed up to 2
0807 03:38:50.974581 (+388156us) log.cc:842] Serialized 14 byte log entry
0807 03:38:51.090170 (+115589us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:38:51.091976 (+ 1806us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:38:51.280897 (+188921us) spinlock_profiling.cc:243] Waited 6.11 ms on lock 0x7b44000232cc. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f1a9a95 00007fe98f1aa4b2 00007fe991e52511 00007fe991e4de1a 00007fe991e41bee 00007fe991e40f36 00007fe991e40b58 00007fe991e4a551 00007fe991e4a509 00007fe991e4a499
0807 03:38:51.752479 (+471582us) raft_consensus.cc:1671] finished
0807 03:38:51.753447 (+ 968us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:38:51.755075 (+ 1628us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":479727,"prepare.queue_time_us":133464,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":28,"raft.queue_time_us":499922,"raft.run_cpu_time_us":276,"raft.run_wall_time_us":10055,"spinlock_wait_cycles":14054144,"thread_start_us":669963,"threads_started":2,"wal-append.queue_time_us":495850}
I20260807 03:38:52.100831 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:52.101842 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:52.102552 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:52.103313 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:52.104568 18295 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57268 (request call id 661) took 1953 ms. Trace:
I20260807 03:38:52.104875 18295 rpcz_store.cc:276] 0807 03:38:50.152457 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:50.237130 (+ 84673us) service_pool.cc:227] Handling call
0807 03:38:50.401464 (+164334us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:38:50.638656 (+237192us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:38:50.639425 (+ 769us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:38:50.642442 (+ 3017us) raft_consensus.cc:1631] Marking committed up to 1
0807 03:38:50.642491 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:38:52.104378 (+1461887us) spinlock_profiling.cc:243] Waited 218 ms on lock 0x7b54000ff854. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:38:52.104417 (+ 39us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:38:52.104525 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":500515712}
I20260807 03:38:52.106081 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:52.107250 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:51.769464 17914 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39494 (request call id 132) took 2611 ms. Trace:
I20260807 03:38:51.853400 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:38:51.863837 18079 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.104s user 0.015s sys 0.019s
I20260807 03:38:51.872661 19650 raft_consensus.cc:482] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
W20260807 03:38:52.110517 18079 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.350s user 0.016s sys 0.019s
I20260807 03:38:52.110639 19650 raft_consensus.cc:504] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.111275 18079 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33268 (request call id 617) took 1372 ms. Trace:
I20260807 03:38:52.111883 18184 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44662 (request call id 437) took 1381 ms. Trace:
I20260807 03:38:52.112818 19650 leader_election.cc:288] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:51.873289 18075 raft_consensus.cc:2932] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:38:51.880718 19724 raft_consensus.cc:482] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:51.882643 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:51.883832 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:51.886516 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:52.111670 18079 rpcz_store.cc:276] 0807 03:38:50.738953 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:50.739488 (+ 535us) service_pool.cc:227] Handling call
0807 03:38:50.739969 (+ 481us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:38:50.757320 (+ 17351us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:38:50.757347 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:38:50.759908 (+ 2561us) log.cc:842] Serialized 26 byte log entry
0807 03:38:52.110945 (+1351037us) raft_consensus.cc:1631] Marking committed up to 1
0807 03:38:52.110984 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:38:52.111065 (+ 81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:38:52.111109 (+ 44us) raft_consensus.cc:1671] finished
0807 03:38:52.111121 (+ 12us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:38:52.111227 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":14575,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":3761,"thread_start_us":1350569,"threads_started":1,"wal-append.queue_time_us":112176,"wal-append.run_cpu_time_us":2022,"wal-append.run_wall_time_us":1002064}
I20260807 03:38:51.886602 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:52.119388 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:52.119793 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:52.108215 17914 rpcz_store.cc:276] 0807 03:38:49.175527 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:49.176219 (+ 692us) service_pool.cc:227] Handling call
0807 03:38:49.930160 (+753941us) spinlock_profiling.cc:243] Waited 137 ms on lock 0x7b54002611d4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144af0 00007fe9921c7f2a 00007fe998119d6c 00007fe998213c3e 00007fe996525fb4 00007fe996525f42 00007fe996525e71 00007fe996525dec 00007fe9965250a2 00007fe9905e994c 00007fe9905e8d86
0807 03:38:51.171177 (+1241017us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:38:51.171206 (+ 29us) write_op.cc:454] Released schema lock
0807 03:38:51.769393 (+598187us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 03:38:50.859904 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 03:38:50.860611 (+ 707us) write_op.cc:432] Acquiring schema lock in shared mode
0807 03:38:50.860658 (+ 47us) write_op.cc:435] Acquired schema lock
0807 03:38:50.860678 (+ 20us) tablet.cc:660] Decoding operations
0807 03:38:50.861513 (+ 835us) write_op.cc:620] Acquiring the partition lock for write op
0807 03:38:50.861583 (+ 70us) write_op.cc:641] Partition lock acquired for write op
0807 03:38:50.861621 (+ 38us) tablet.cc:683] Acquiring locks for 4 operations
0807 03:38:50.862170 (+ 549us) tablet.cc:699] Row locks acquired
0807 03:38:50.862195 (+ 25us) write_op.cc:260] PREPARE: finished
0807 03:38:50.862387 (+ 192us) write_op.cc:270] Start()
0807 03:38:50.862515 (+ 128us) write_op.cc:276] Timestamp: P: 1786073930862332 usec, L: 0
0807 03:38:50.862545 (+ 30us) op_driver.cc:331] REPLICATION: starting
0807 03:38:50.863341 (+ 796us) log.cc:842] Serialized 1776 byte log entry
0807 03:38:51.001726 (+138385us) op_driver.cc:444] REPLICATION: finished
0807 03:38:51.054824 (+ 53098us) write_op.cc:301] APPLY: starting
0807 03:38:51.054903 (+ 79us) tablet.cc:1365] starting BulkCheckPresence
0807 03:38:51.055066 (+ 163us) tablet.cc:1368] finished BulkCheckPresence
0807 03:38:51.055556 (+ 490us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 03:38:51.056546 (+ 990us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 03:38:51.056717 (+ 171us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0807 03:38:51.056760 (+ 43us) write_op.cc:312] APPLY: finished
0807 03:38:51.062561 (+ 5801us) log.cc:842] Serialized 40 byte log entry
0807 03:38:51.062723 (+ 162us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:38:51.062809 (+ 86us) write_op.cc:454] Released schema lock
0807 03:38:51.062845 (+ 36us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":44599,"reported_tablets":4,"spinlock_wait_cycles":314164992,"child_traces":[["op",{"apply.queue_time_us":52844,"apply.run_cpu_time_us":2430,"apply.run_wall_time_us":8365,"num_ops":4,"prepare.queue_time_us":191169,"prepare.run_cpu_time_us":31432,"prepare.run_wall_time_us":49882,"raft.queue_time_us":35703,"raft.run_cpu_time_us":15931,"raft.run_wall_time_us":53699,"replication_time_us":139071,"thread_start_us":645953,"threads_started":4,"wal-append.queue_time_us":69820,"wal-append.run_cpu_time_us":8313,"wal-append.run_wall_time_us":1132399}]]}
I20260807 03:38:51.897701 19578 raft_consensus.cc:482] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
W20260807 03:38:51.901821 19698 log.cc:924] Time spent T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.688s user 0.000s sys 0.010s
I20260807 03:38:51.904841 18081 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33268 (request call id 611) took 2186 ms. Trace:
I20260807 03:38:51.953696 19612 raft_consensus.cc:482] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:51.957163 18140 leader_election.cc:302] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:51.966569 19583 raft_consensus.cc:482] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:51.981745 19719 raft_consensus.cc:482] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:51.985302 18249 leader_election.cc:302] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:51.986727 18187 raft_consensus.cc:1226] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 2. Current term is 3. Ops: []
I20260807 03:38:51.990209 19617 raft_consensus.cc:482] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:51.997263 17915 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35098 (request call id 18) took 1249 ms. Trace:
I20260807 03:38:52.112475 18184 rpcz_store.cc:276] 0807 03:38:50.730413 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:50.735762 (+ 5349us) service_pool.cc:227] Handling call
0807 03:38:50.736131 (+ 369us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:38:50.785657 (+ 49526us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:38:50.788278 (+ 2621us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:38:50.793017 (+ 4739us) raft_consensus.cc:1631] Marking committed up to 1
0807 03:38:50.794102 (+ 1085us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:38:52.111666 (+1317564us) spinlock_profiling.cc:243] Waited 44.2 ms on lock 0x7b54003c9e54. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:38:52.111709 (+ 43us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:38:52.111822 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":101635456}
I20260807 03:38:52.003376 19659 raft_consensus.cc:482] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:52.003638 19488 raft_consensus.cc:482] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:52.012401 18188 raft_consensus.cc:1226] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:52.019040 19641 raft_consensus.cc:482] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
W20260807 03:38:52.025182 18300 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57268 (request call id 666) took 1794 ms (client timeout 1565 ms). Trace:
I20260807 03:38:52.029506 19728 raft_consensus.cc:482] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:52.030923 19292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:38:52.041306 18185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44662 (request call id 436) took 1311 ms. Trace:
I20260807 03:38:52.052060 19726 raft_consensus.cc:482] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:52.058101 19456 raft_consensus.cc:2781] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260807 03:38:51.448342 19711 log.cc:924] Time spent T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.165s user 0.000s sys 0.001s
I20260807 03:38:51.262362 19594 raft_consensus.cc:2726] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 03:38:52.069169 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:52.168241 19714 raft_consensus.cc:2726] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:52.070410 19479 raft_consensus.cc:2726] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:52.070787 19729 raft_consensus.cc:2781] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:52.071060 18248 leader_election.cc:302] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
W20260807 03:38:52.071973 19544 consensus_meta.cc:287] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 0.850s user 0.012s sys 0.010s
I20260807 03:38:52.080881 19608 raft_consensus.cc:482] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:52.221901 19729 raft_consensus.cc:482] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:52.223696 19729 raft_consensus.cc:3037] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:52.223381 19544 raft_consensus.cc:504] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.223518 19608 raft_consensus.cc:504] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.081423 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:52.225500 19544 leader_election.cc:288] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:52.225652 18312 raft_consensus.cc:2454] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:38:52.225831 19608 leader_election.cc:288] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:52.173300 19617 raft_consensus.cc:504] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.168359 19612 raft_consensus.cc:504] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.085099 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:52.227252 18141 leader_election.cc:302] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:52.228147 19544 raft_consensus.cc:2781] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:38:52.228241 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:52.228580 19544 raft_consensus.cc:482] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:52.229041 19544 raft_consensus.cc:3037] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:38:52.085188 19496 raft_consensus.cc:482] T af8877c3fbba4ab18765832f1390921f P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:52.083741 19388 raft_consensus.cc:482] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:52.085991 19480 raft_consensus.cc:482] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:52.087678 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:52.088780 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
W20260807 03:38:52.222417 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 151434us
I20260807 03:38:52.230659 18248 leader_election.cc:302] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:52.231761 18248 leader_election.cc:302] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:52.114418 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:52.233712 18073 raft_consensus.cc:2454] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
W20260807 03:38:52.234390 18073 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33268 (request call id 625) took 1483 ms (client timeout 1601 ms). Trace:
W20260807 03:38:52.234815 18073 rpcz_store.cc:269] 0807 03:38:50.750903 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:50.759156 (+ 8253us) service_pool.cc:227] Handling call
0807 03:38:52.114309 (+1355153us) spinlock_profiling.cc:243] Waited 4.12 ms on lock 0x7b5800001298. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f25f851 00007fe98f25f702 00007fe99814fb09 00007fe99708b730 00007fe99715179b 00007fe997151821 00007fe9971011b0 00007fe997100a94 00007fe9907e8c04 00007fe9907e8b92
0807 03:38:52.234289 (+119980us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9471232}
I20260807 03:38:52.115048 18191 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44662 (request call id 435) took 1385 ms. Trace:
I20260807 03:38:52.166460 19578 raft_consensus.cc:504] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.156828 18079 raft_consensus.cc:2454] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:52.155109 18298 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57276 (request call id 440) took 1523 ms. Trace:
I20260807 03:38:52.235669 19729 raft_consensus.cc:504] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.165234 18075 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33268 (request call id 614) took 1432 ms. Trace:
W20260807 03:38:52.166926 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 209847us
W20260807 03:38:52.167236 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 101960us
I20260807 03:38:52.164534 18081 rpcz_store.cc:276] 0807 03:38:49.718071 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:49.730031 (+ 11960us) service_pool.cc:227] Handling call
0807 03:38:49.730424 (+ 393us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:38:50.090961 (+360537us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:38:50.091009 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:38:50.093644 (+ 2635us) log.cc:842] Serialized 26 byte log entry
0807 03:38:50.251380 (+157736us) raft_consensus.cc:1631] Marking committed up to 2
0807 03:38:50.627506 (+376126us) log.cc:842] Serialized 14 byte log entry
0807 03:38:50.648249 (+ 20743us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:38:50.648342 (+ 93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:38:51.904626 (+1256284us) raft_consensus.cc:1671] finished
0807 03:38:51.904646 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:38:51.904765 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":36302,"prepare.queue_time_us":476768,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":396131,"raft.run_cpu_time_us":335,"raft.run_wall_time_us":16320,"thread_start_us":311924,"threads_started":2,"wal-append.queue_time_us":461176,"wal-append.run_cpu_time_us":11393,"wal-append.run_wall_time_us":1462793}
I20260807 03:38:52.169268 17915 rpcz_store.cc:276] 0807 03:38:50.747453 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:50.747614 (+ 161us) service_pool.cc:227] Handling call
0807 03:38:51.997201 (+1249587us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1}
I20260807 03:38:52.001976 19490 raft_consensus.cc:482] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
W20260807 03:38:52.170228 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 185005us
I20260807 03:38:52.244290 18027 leader_election.cc:302] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:52.244601 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:52.244549 19490 raft_consensus.cc:504] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.245496 18027 leader_election.cc:302] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:52.245774 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:52.246665 19490 leader_election.cc:288] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:52.247205 19490 raft_consensus.cc:2726] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:52.247234 18027 leader_election.cc:302] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
W20260807 03:38:52.172117 18300 rpcz_store.cc:269] 0807 03:38:50.231004 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:50.231212 (+ 208us) service_pool.cc:227] Handling call
0807 03:38:52.024900 (+1793688us) spinlock_profiling.cc:243] Waited 256 ms on lock 0x7b5400262ad4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:38:52.025052 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":588773760}
I20260807 03:38:52.248502 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:52.249044 19650 raft_consensus.cc:2726] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:52.249750 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:52.249920 19479 raft_consensus.cc:2726] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:52.167306 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:52.216303 19645 leader_election.cc:288] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:52.169772 19583 raft_consensus.cc:504] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.252393 19583 leader_election.cc:288] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:52.170513 19469 raft_consensus.cc:482] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:52.220664 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:52.253050 18070 raft_consensus.cc:2454] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 1.
I20260807 03:38:52.222862 19471 raft_consensus.cc:2726] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:52.236276 18191 rpcz_store.cc:276] 0807 03:38:50.729822 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:51.853062 (+1123240us) service_pool.cc:227] Handling call
0807 03:38:52.114978 (+261916us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 03:38:52.173314 19659 raft_consensus.cc:504] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.081688 19291 raft_consensus.cc:1226] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 2 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:52.168942 19724 raft_consensus.cc:504] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.174393 19705 raft_consensus.cc:3032] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 2 LEADER]: Stepping down as leader of term 2
I20260807 03:38:52.104061 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:52.252948 19469 raft_consensus.cc:504] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.237192 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:52.239114 19388 raft_consensus.cc:504] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.173238 19604 raft_consensus.cc:2726] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 03:38:52.221909 19726 raft_consensus.cc:504] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.174335 19641 raft_consensus.cc:504] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.164810 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:52.244906 18249 leader_election.cc:302] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:52.269402 18080 raft_consensus.cc:2396] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 1.
I20260807 03:38:52.172854 18188 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44660 (request call id 739) took 1184 ms. Trace:
I20260807 03:38:52.249768 19650 raft_consensus.cc:2726] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:52.252291 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:52.261621 19705 raft_consensus.cc:729] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:38:52.253901 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:52.221987 18071 raft_consensus.cc:2362] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3. Re-sending same reply.
I20260807 03:38:52.170387 19719 raft_consensus.cc:504] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.255285 18140 leader_election.cc:302] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:52.240778 19608 raft_consensus.cc:2726] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:52.285772 18140 leader_election.cc:302] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:52.286739 19714 raft_consensus.cc:2781] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:52.286819 19490 consensus_queue.cc:1046] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:38:52.240780 18298 rpcz_store.cc:276] 0807 03:38:50.631642 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:50.870459 (+238817us) service_pool.cc:227] Handling call
0807 03:38:52.025818 (+1155359us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:38:52.026073 (+ 255us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:38:52.026113 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:38:52.026179 (+ 66us) raft_consensus.cc:1631] Marking committed up to 1
0807 03:38:52.026225 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:38:52.026302 (+ 77us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:38:52.154913 (+128611us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b5400262ad0. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bea09 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:38:52.155041 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2656183168}
I20260807 03:38:52.289180 18140 leader_election.cc:302] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:52.251811 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:52.290364 19625 raft_consensus.cc:2726] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:52.291483 18140 leader_election.cc:302] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:52.292219 19625 raft_consensus.cc:2781] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:52.292299 19291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:38:52.292615 19625 raft_consensus.cc:482] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:52.293074 19625 raft_consensus.cc:3037] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:52.294317 18025 leader_election.cc:302] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:52.241712 19480 raft_consensus.cc:504] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.295785 19650 raft_consensus.cc:2726] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:52.297449 19480 leader_election.cc:288] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:52.298295 19612 leader_election.cc:288] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:52.298825 19612 raft_consensus.cc:3032] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:52.299175 19612 raft_consensus.cc:729] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:52.303357 19625 raft_consensus.cc:504] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.239547 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:52.270022 18188 rpcz_store.cc:276] 0807 03:38:50.988642 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:51.036241 (+ 47599us) service_pool.cc:227] Handling call
0807 03:38:52.012037 (+975796us) spinlock_profiling.cc:243] Waited 4.53 ms on lock 0x7b5800003998. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f25f851 00007fe98f25f702 00007fe99814fb09 00007fe99708b730 00007fe99715179b 00007fe997151821 00007fe9971152b0 00007fe9971007f9 00007fe9971002a2 00007fe9907e6684
0807 03:38:52.012238 (+ 201us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:38:52.172678 (+160440us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:38:52.172813 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10414976}
I20260807 03:38:52.252547 18027 leader_election.cc:302] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:52.240550 19496 raft_consensus.cc:504] T af8877c3fbba4ab18765832f1390921f P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:38:52.263578 19594 raft_consensus.cc:482] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:52.265995 19456 raft_consensus.cc:482] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:52.346256 18140 leader_election.cc:302] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:52.280695 19705 consensus_queue.cc:260] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.273774 19471 raft_consensus.cc:2781] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:38:52.281677 19578 leader_election.cc:288] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:52.208959 18185 rpcz_store.cc:276] 0807 03:38:50.730122 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:50.756159 (+ 26037us) service_pool.cc:227] Handling call
0807 03:38:50.764797 (+ 8638us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:38:50.765320 (+ 523us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:38:50.765365 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:38:50.766017 (+ 652us) log.cc:842] Serialized 26 byte log entry
0807 03:38:51.773664 (+1007647us) raft_consensus.cc:1631] Marking committed up to 1
0807 03:38:51.773730 (+ 66us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:38:51.773807 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:38:52.040607 (+266800us) spinlock_profiling.cc:243] Waited 849 ms on lock 0x7b54000df0a0. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe99214d771 00007fe992158220 00007fe9921581c8 00007fe992158148 00007fe992158103 00007fe99215738c 00007fe9981c8206 00007fe9981c0808 00007fe99219438b 00007fe992196e80
0807 03:38:52.040953 (+ 346us) raft_consensus.cc:1671] finished
0807 03:38:52.040983 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:38:52.041235 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1424,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":138,"spinlock_wait_cycles":1952586496,"thread_start_us":66010,"threads_started":1,"wal-append.queue_time_us":140591}
I20260807 03:38:52.372972 19469 leader_election.cc:288] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:52.373063 19705 raft_consensus.cc:3037] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:52.282675 19388 leader_election.cc:288] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:52.282966 19729 leader_election.cc:288] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:52.373112 19480 raft_consensus.cc:2726] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:38:52.284196 19726 leader_election.cc:288] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:52.287166 19714 raft_consensus.cc:482] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:52.289323 19719 leader_election.cc:288] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:52.180198 19488 raft_consensus.cc:504] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.292697 19724 leader_election.cc:288] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:52.242167 18075 rpcz_store.cc:276] 0807 03:38:50.732457 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:50.735618 (+ 3161us) service_pool.cc:227] Handling call
0807 03:38:50.736192 (+ 574us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:38:50.736504 (+ 312us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:38:50.736544 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:38:50.756601 (+ 20057us) log.cc:842] Serialized 572 byte log entry
0807 03:38:51.873187 (+1116586us) raft_consensus.cc:1631] Marking committed up to 3
0807 03:38:52.164698 (+291511us) log.cc:842] Serialized 14 byte log entry
0807 03:38:52.164918 (+ 220us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:38:52.164998 (+ 80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:38:52.165035 (+ 37us) raft_consensus.cc:1671] finished
0807 03:38:52.165045 (+ 10us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:38:52.165181 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":51160,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":156,"spinlock_wait_cycles":189312,"thread_start_us":3285,"threads_started":1,"wal-append.queue_time_us":1106783}
I20260807 03:38:52.173843 19728 raft_consensus.cc:504] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.171732 19495 raft_consensus.cc:2781] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 03:38:52.299270 18082 raft_consensus.cc:2454] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
I20260807 03:38:52.311475 19612 consensus_queue.cc:260] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:38:52.379894 19495 raft_consensus.cc:686] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
W20260807 03:38:52.338124 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 85757.4us
I20260807 03:38:52.380905 18027 leader_election.cc:302] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:52.380805 19495 consensus_queue.cc:237] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.381982 19496 leader_election.cc:288] T af8877c3fbba4ab18765832f1390921f P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:38:52.364480 19594 raft_consensus.cc:504] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
W20260807 03:38:52.344982 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 77741.2us
I20260807 03:38:52.384841 19659 leader_election.cc:288] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:52.385013 18248 leader_election.cc:302] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:52.385179 19594 leader_election.cc:288] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:38:52.385733 19594 raft_consensus.cc:2781] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:52.386148 19469 raft_consensus.cc:2726] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:38:52.386518 19291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:38:52.386598 18248 leader_election.cc:302] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:52.387518 19469 raft_consensus.cc:2726] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:52.377276 18071 raft_consensus.cc:1226] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:52.388398 19617 leader_election.cc:288] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:52.389608 19488 leader_election.cc:288] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:52.391593 19641 leader_election.cc:288] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:52.392697 19544 raft_consensus.cc:504] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.394701 19544 leader_election.cc:288] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:52.387910 18248 leader_election.cc:302] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:52.377612 19471 raft_consensus.cc:686] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:52.395895 19388 raft_consensus.cc:2726] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:52.396430 19471 consensus_queue.cc:237] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.399422 18248 leader_election.cc:302] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:38:52.386072 19594 raft_consensus.cc:482] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:52.400585 19594 raft_consensus.cc:3037] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:52.400766 19388 raft_consensus.cc:2781] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:52.401034 19625 leader_election.cc:288] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:52.401207 19388 raft_consensus.cc:482] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:52.401582 19388 raft_consensus.cc:3037] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
W20260807 03:38:52.353538 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 95015.5us
I20260807 03:38:52.364295 19456 raft_consensus.cc:3037] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:38:52.345968 19650 raft_consensus.cc:2726] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:52.283933 19583 raft_consensus.cc:2726] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 03:38:52.374902 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 102347us
I20260807 03:38:52.374321 19714 raft_consensus.cc:3037] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:52.379343 19728 leader_election.cc:288] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:52.405424 18248 leader_election.cc:302] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
W20260807 03:38:52.406869 18027 connection.cc:232] RPC call timeout handler was delayed by -1.2860561180859804s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1543s for a response
W20260807 03:38:52.415751 18027 connection.cc:232] RPC call timeout handler was delayed by -1.1717561180703342s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.167s for a response
I20260807 03:38:52.415951 18248 leader_election.cc:302] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
W20260807 03:38:52.417127 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 70957.6us
I20260807 03:38:52.418861 18248 leader_election.cc:302] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:52.420339 18248 leader_election.cc:302] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:52.414364 18079 raft_consensus.cc:2932] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:38:52.423822 18248 leader_election.cc:302] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:52.422685 19488 raft_consensus.cc:2726] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:52.429100 19617 raft_consensus.cc:2726] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:38:52.422617 19747 raft_consensus.cc:2726] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:52.429404 19659 raft_consensus.cc:2726] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:38:52.441258 18140 leader_election.cc:302] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:52.443114 19388 raft_consensus.cc:504] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:52.444211 18246 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.132:41381, user_credentials={real_user=slave}} blocked reactor thread for 132633us
I20260807 03:38:52.446163 19612 raft_consensus.cc:3037] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:52.428205 19743 raft_consensus.cc:2726] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:52.450176 19456 raft_consensus.cc:504] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.458649 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:52.459318 18295 raft_consensus.cc:3037] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:38:52.459203 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:52.460218 18305 raft_consensus.cc:2454] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:52.461232 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:52.462327 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:52.460247 19488 raft_consensus.cc:482] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:52.463557 19291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:38:52.463639 19488 raft_consensus.cc:504] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.465377 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:52.459857 18310 raft_consensus.cc:1226] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 3. Current term is 4. Ops: []
I20260807 03:38:52.468461 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:52.470789 18305 raft_consensus.cc:1226] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260807 03:38:52.471964 18305 raft_consensus.cc:1226] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:52.464993 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:52.473604 19488 leader_election.cc:288] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:52.459167 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:52.474752 18309 raft_consensus.cc:2379] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 2: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:38:52.464509 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:52.476833 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:52.481724 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:52.483459 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:52.483677 19388 leader_election.cc:288] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:52.484560 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:52.485113 18309 raft_consensus.cc:2454] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:38:52.485136 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:52.475914 17915 catalog_manager.cc:5495] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 reported cstate change: config changed from index 2 to 3, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:38:52.492180 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:52.491755 17915 catalog_manager.cc:5495] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 reported cstate change: config changed from index 2 to 3, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
W20260807 03:38:52.497217 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 56524.9us
I20260807 03:38:52.499923 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:52.493929 17915 catalog_manager.cc:5495] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 reported cstate change: config changed from index 2 to 3, term changed from 1 to 2, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to <none>, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:38:52.503420 19728 consensus_queue.cc:1046] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:52.504984 19728 consensus_queue.cc:1225] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer 66a36e1ac8e945e893668e282d394c49 log is divergent from this leader: its last log entry 4.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260807 03:38:52.509598 18027 leader_election.cc:302] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:52.510197 18248 leader_election.cc:302] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:52.505487 19728 raft_consensus.cc:3032] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:38:52.513705 18248 leader_election.cc:302] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:38:52.514801 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:52.517151 19728 raft_consensus.cc:729] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:38:52.490782 19456 leader_election.cc:288] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:52.518245 19745 raft_consensus.cc:2726] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:52.518066 19728 consensus_queue.cc:260] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.519246 19728 raft_consensus.cc:3037] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:38:52.518198 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:52.522346 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:52.523586 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:52.524224 18311 raft_consensus.cc:2454] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:38:52.524893 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:52.525477 18081 raft_consensus.cc:2454] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:38:52.527552 18081 raft_consensus.cc:1261] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:38:52.529810 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:52.532349 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:52.512568 19594 raft_consensus.cc:504] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.535246 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:52.536427 18300 raft_consensus.cc:2396] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for term 2 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20260807 03:38:52.505844 19625 raft_consensus.cc:2781] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:52.538254 19625 raft_consensus.cc:482] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:52.538640 19625 raft_consensus.cc:3037] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:52.535748 19594 leader_election.cc:288] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:52.542810 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:52.523999 19714 raft_consensus.cc:504] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.545537 18027 leader_election.cc:302] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:52.546041 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:52.546353 18248 leader_election.cc:302] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:52.548527 18248 leader_election.cc:302] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:38:52.551755 18027 leader_election.cc:302] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:52.553314 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:52.554601 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:52.557214 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:52.557777 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:52.505797 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:52.562251 19729 raft_consensus.cc:482] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:52.527161 19705 raft_consensus.cc:2781] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:52.556550 19641 raft_consensus.cc:2726] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:52.570607 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:52.571380 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ca0223f47b46fd955704eba5259370" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:52.572376 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:52.574946 18079 raft_consensus.cc:2454] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:52.575136 18073 raft_consensus.cc:2454] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:52.572455 18191 raft_consensus.cc:1226] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:52.568608 18075 raft_consensus.cc:1226] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 2. Current term is 3. Ops: []
I20260807 03:38:52.576776 18027 leader_election.cc:302] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:52.577132 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:38:52.573096 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 141134us
I20260807 03:38:52.555606 19659 raft_consensus.cc:2726] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:52.578258 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:52.578469 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:52.578902 18193 raft_consensus.cc:1226] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:52.579109 18190 raft_consensus.cc:2379] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 2: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:38:52.562915 19742 raft_consensus.cc:2781] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:38:52.583424 18073 raft_consensus.cc:2454] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:52.581561 18190 raft_consensus.cc:1226] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 3. Current term is 4. Ops: []
I20260807 03:38:52.585043 18188 raft_consensus.cc:1226] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:52.544732 19724 consensus_queue.cc:1046] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20260807 03:38:52.588537 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:52.582140 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:52.562809 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:52.559020 19714 leader_election.cc:288] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:52.563599 19594 raft_consensus.cc:2781] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:52.585085 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:52.596726 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:52.577625 18295 raft_consensus.cc:2454] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:38:52.590948 18193 raft_consensus.cc:1226] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 3. Current term is 4. Ops: []
I20260807 03:38:52.590708 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:52.585778 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:52.563148 19608 consensus_queue.cc:1046] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20260807 03:38:52.566805 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:52.561131 19650 consensus_queue.cc:1046] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20260807 03:38:52.562754 19729 raft_consensus.cc:504] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.606166 19625 raft_consensus.cc:504] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.590513 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:52.609020 18071 raft_consensus.cc:2379] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 2: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:38:52.614092 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:52.596086 19594 raft_consensus.cc:482] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:52.615504 19594 raft_consensus.cc:3037] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:52.567430 19705 raft_consensus.cc:482] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:52.603768 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:52.616537 19705 raft_consensus.cc:3037] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:52.617205 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:52.601467 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:52.617849 18070 raft_consensus.cc:2454] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:52.600575 19743 raft_consensus.cc:2741] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
I20260807 03:38:52.607162 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:52.606973 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:52.611984 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:52.599274 19726 raft_consensus.cc:2726] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:52.605237 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:52.612711 18073 raft_consensus.cc:2454] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:52.615386 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:52.611415 18189 raft_consensus.cc:1226] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 2. Current term is 3. Ops: []
I20260807 03:38:52.617237 19650 raft_consensus.cc:3032] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:38:52.612408 19729 leader_election.cc:288] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:52.623473 19650 raft_consensus.cc:729] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:38:52.624296 19650 consensus_queue.cc:260] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.625463 19650 raft_consensus.cc:3037] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:38:52.628382 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:52.612967 19625 leader_election.cc:288] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:52.631283 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:52.631712 19490 raft_consensus.cc:3032] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:52.632081 19490 raft_consensus.cc:729] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:52.610441 18082 raft_consensus.cc:3037] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:38:52.632969 19490 consensus_queue.cc:260] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.634191 19490 raft_consensus.cc:3037] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:52.634286 19641 raft_consensus.cc:3032] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:52.634819 19641 raft_consensus.cc:729] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:38:52.618609 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:52.640224 19594 raft_consensus.cc:504] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.641496 18071 raft_consensus.cc:2379] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 2: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:38:52.597718 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:52.635782 19641 consensus_queue.cc:260] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.644757 19641 raft_consensus.cc:3037] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:52.645797 18082 raft_consensus.cc:2454] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:38:52.650413 19705 raft_consensus.cc:504] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.650764 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:52.651486 18081 raft_consensus.cc:1226] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:52.652539 18080 raft_consensus.cc:2396] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 03:38:52.652808 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:52.598340 18027 leader_election.cc:302] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:52.655645 19729 raft_consensus.cc:2726] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:38:52.658574 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
W20260807 03:38:52.660012 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 61761.2us
I20260807 03:38:52.662468 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:52.666467 18249 leader_election.cc:302] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:52.669452 19743 raft_consensus.cc:2726] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:38:52.668521 18027 leader_election.cc:302] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:52.671124 18249 leader_election.cc:302] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:52.675529 18081 raft_consensus.cc:1226] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:52.678916 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:52.672845 18027 leader_election.cc:302] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:52.680328 19729 raft_consensus.cc:2726] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:52.602324 19765 raft_consensus.cc:482] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:52.681931 18249 leader_election.cc:302] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:52.686738 19659 raft_consensus.cc:2781] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 03:38:52.687227 19659 raft_consensus.cc:686] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:52.682111 18027 leader_election.cc:302] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:52.690569 19729 raft_consensus.cc:2781] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:38:52.691161 19729 raft_consensus.cc:686] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:38:52.692147 19729 consensus_queue.cc:237] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.692639 19469 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:52.693125 19469 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.692353 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:52.695120 19469 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:52.697340 18248 leader_election.cc:302] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:52.700274 18027 leader_election.cc:302] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:52.700721 19469 raft_consensus.cc:2781] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:52.701707 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:52.702666 19726 raft_consensus.cc:2726] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:38:52.703227 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:52.703390 18027 leader_election.cc:302] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:52.704433 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:52.705474 19496 raft_consensus.cc:482] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:52.705945 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:52.706128 19496 raft_consensus.cc:504] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.707839 19479 raft_consensus.cc:2726] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:52.709403 18027 leader_election.cc:302] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:52.700639 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ca0223f47b46fd955704eba5259370" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:52.710106 19724 raft_consensus.cc:3037] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:52.711145 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:52.709266 19645 consensus_queue.cc:1046] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:52.712939 19645 raft_consensus.cc:3032] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:52.713368 19645 raft_consensus.cc:729] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:38:52.713410 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:52.714519 19645 consensus_queue.cc:260] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.715067 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:52.701618 18248 leader_election.cc:302] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:52.720432 18188 raft_consensus.cc:1226] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:52.721933 18027 leader_election.cc:302] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:52.725068 18082 raft_consensus.cc:1226] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:52.729655 18027 leader_election.cc:302] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:52.718439 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:52.702288 19469 raft_consensus.cc:482] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:52.709810 19496 leader_election.cc:288] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:52.743607 18248 leader_election.cc:302] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:52.681339 19728 raft_consensus.cc:2781] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:52.750212 18248 leader_election.cc:302] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:52.749396 18140 leader_election.cc:302] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:52.751745 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
W20260807 03:38:52.742909 19775 log.cc:924] Time spent T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.057s user 0.001s sys 0.000s
I20260807 03:38:52.732857 19479 raft_consensus.cc:2726] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:52.735870 19578 raft_consensus.cc:2726] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 03:38:52.735920 19645 raft_consensus.cc:3037] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:52.745317 18027 leader_election.cc:302] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:52.757128 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:52.757535 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:52.760747 18300 raft_consensus.cc:1226] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:52.745615 19469 raft_consensus.cc:3037] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:52.736429 19488 raft_consensus.cc:2781] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:52.777032 19742 raft_consensus.cc:686] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [term 2 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:52.733495 19772 raft_consensus.cc:2781] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:38:52.689044 19773 consensus_queue.cc:1046] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.056s
I20260807 03:38:52.751307 19728 raft_consensus.cc:482] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:52.753460 19496 raft_consensus.cc:2781] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:52.750082 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:52.789201 18249 leader_election.cc:302] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:52.787539 18190 raft_consensus.cc:1226] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:52.792768 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:38:52.750237 18223 debug-util.cc:398] Leaking SignalData structure 0x7b080016b800 after lost signal to thread 16157
I20260807 03:38:52.773855 19490 raft_consensus.cc:2781] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:52.736028 19726 consensus_queue.cc:1046] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.038s
I20260807 03:38:52.798045 18249 leader_election.cc:302] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:52.733659 19765 raft_consensus.cc:504] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.786707 19479 raft_consensus.cc:2726] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:52.803066 18249 leader_election.cc:302] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:52.786394 19604 raft_consensus.cc:2726] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:52.785019 19488 raft_consensus.cc:482] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:52.816560 19488 raft_consensus.cc:3037] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:52.792307 19728 raft_consensus.cc:3037] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:52.792358 19496 raft_consensus.cc:482] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:52.826407 19496 raft_consensus.cc:3037] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:52.829006 19743 raft_consensus.cc:482] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:52.829516 19743 raft_consensus.cc:504] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.793853 19772 raft_consensus.cc:686] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 2 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:52.736346 19659 consensus_queue.cc:237] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.832572 18249 leader_election.cc:302] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:52.832839 18191 raft_consensus.cc:2454] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:38:52.801059 19490 raft_consensus.cc:482] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:52.813644 18189 raft_consensus.cc:2379] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 2: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:38:52.844743 19594 leader_election.cc:288] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:52.838135 19471 raft_consensus.cc:3037] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:52.834052 18249 leader_election.cc:302] T af8877c3fbba4ab18765832f1390921f P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:38:52.809348 19388 raft_consensus.cc:2726] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:52.812863 19608 raft_consensus.cc:2726] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:38:52.812376 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:52.799537 19617 consensus_queue.cc:1046] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.038s
I20260807 03:38:52.813915 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:52.853665 19789 raft_consensus.cc:2726] T af8877c3fbba4ab18765832f1390921f P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 03:38:52.860967 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 104452us
I20260807 03:38:52.855576 18249 leader_election.cc:302] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:52.858101 19471 raft_consensus.cc:2726] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 03:38:52.848470 18223 debug-util.cc:398] Leaking SignalData structure 0x7b0800171ec0 after lost signal to thread 17976
W20260807 03:38:52.861562 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 84742.2us
I20260807 03:38:52.837453 19768 raft_consensus.cc:2726] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:52.837571 19705 leader_election.cc:288] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:52.846297 19490 raft_consensus.cc:3037] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:52.877790 18140 leader_election.cc:302] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:52.886669 19496 raft_consensus.cc:504] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.887734 18309 raft_consensus.cc:1226] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:52.890818 19743 leader_election.cc:288] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:52.891140 19724 raft_consensus.cc:2726] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:52.891829 19772 consensus_queue.cc:237] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.894243 18249 leader_election.cc:302] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:52.872349 19608 raft_consensus.cc:2726] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:52.895941 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:52.897763 18075 raft_consensus.cc:2379] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 2: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:38:52.897866 18140 leader_election.cc:302] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:52.898609 19795 raft_consensus.cc:2726] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:52.899096 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:52.901713 18140 leader_election.cc:302] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:52.899785 18071 raft_consensus.cc:2454] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:38:52.906121 19496 leader_election.cc:288] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:52.900607 19743 raft_consensus.cc:3037] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:52.910667 19456 raft_consensus.cc:2726] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 03:38:52.906813 19795 raft_consensus.cc:2726] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 03:38:52.912251 19765 leader_election.cc:288] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:52.914024 19469 raft_consensus.cc:504] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.914978 19729 raft_consensus.cc:3032] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:52.879674 19769 consensus_queue.cc:1035] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20260807 03:38:52.921502 19729 raft_consensus.cc:729] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:38:52.922335 19729 consensus_queue.cc:260] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.921178 18249 leader_election.cc:302] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:52.924726 19729 raft_consensus.cc:3037] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:52.925314 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:52.924539 18140 leader_election.cc:302] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:52.927331 18140 leader_election.cc:302] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:52.900943 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:52.929410 18248 leader_election.cc:302] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
W20260807 03:38:52.926481 18223 debug-util.cc:398] Leaking SignalData structure 0x7b0800262aa0 after lost signal to thread 18113
I20260807 03:38:52.897183 19612 consensus_queue.cc:1046] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20260807 03:38:52.928900 19492 raft_consensus.cc:2726] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 03:38:52.876474 19789 consensus_queue.cc:1046] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.054s
I20260807 03:38:52.860822 19742 consensus_queue.cc:237] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.905431 19495 consensus_queue.cc:1046] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20260807 03:38:52.938989 18191 raft_consensus.cc:1261] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260807 03:38:52.940829 18140 leader_election.cc:302] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:52.932974 19719 raft_consensus.cc:2781] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:38:52.943226 18140 leader_election.cc:302] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:52.942327 18248 leader_election.cc:302] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:52.945931 19768 raft_consensus.cc:3032] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:52.946324 19768 raft_consensus.cc:729] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:52.947059 19768 consensus_queue.cc:260] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:52.948151 19471 consensus_queue.cc:1159] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "3c4bceb836db4536a0f255fdf86abc02" responder_term: 1 status { last_received { term: 1 index: 1 } last_committed_idx: 1 last_received_current_leader { term: 1 index: 1 } }
I20260807 03:38:52.949728 19488 raft_consensus.cc:504] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.950665 19617 raft_consensus.cc:3032] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:52.951146 19617 raft_consensus.cc:729] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:52.951721 19488 leader_election.cc:288] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:52.952888 19768 raft_consensus.cc:3037] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:52.952226 19617 consensus_queue.cc:260] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:52.934022 19769 raft_consensus.cc:2726] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 03:38:52.956403 19492 raft_consensus.cc:2726] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:38:52.930572 19799 raft_consensus.cc:2726] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 03:38:52.958600 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ca0223f47b46fd955704eba5259370" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:52.960726 18080 raft_consensus.cc:3037] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:52.960335 19495 raft_consensus.cc:2726] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:52.958786 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:52.965435 19469 leader_election.cc:288] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:52.965560 18072 raft_consensus.cc:3037] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:52.953594 19719 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:52.970475 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:52.981277 19769 consensus_queue.cc:1046] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 03:38:52.969507 19794 consensus_queue.cc:1046] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 03:38:52.982761 19719 raft_consensus.cc:3037] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
W20260807 03:38:52.973659 18223 debug-util.cc:398] Leaking SignalData structure 0x7b0800167c60 after lost signal to thread 18228
I20260807 03:38:52.987229 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ca0223f47b46fd955704eba5259370" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:52.988251 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:52.989540 19773 raft_consensus.cc:3032] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 03:38:52.957931 19803 raft_consensus.cc:2781] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader election won for term 4
W20260807 03:38:52.997960 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 201484us
I20260807 03:38:52.990509 19492 raft_consensus.cc:3032] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:53.005131 18070 raft_consensus.cc:1226] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:53.006673 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:38:53.006754 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 271114us
I20260807 03:38:53.008415 18081 raft_consensus.cc:1261] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 03:38:52.991328 19617 raft_consensus.cc:3037] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:53.009418 18249 leader_election.cc:302] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:53.009742 18027 leader_election.cc:302] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:53.014644 18080 raft_consensus.cc:2454] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
I20260807 03:38:53.014794 18027 leader_election.cc:302] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:53.015249 18073 raft_consensus.cc:1226] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:53.008080 19495 consensus_queue.cc:1035] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20260807 03:38:53.002319 19492 raft_consensus.cc:729] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:52.996178 19773 raft_consensus.cc:729] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:53.018834 18027 leader_election.cc:302] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:53.017395 18191 raft_consensus.cc:1261] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 03:38:53.021026 19650 raft_consensus.cc:2726] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 03:38:53.022497 19544 raft_consensus.cc:3032] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:53.019685 18248 leader_election.cc:302] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:38:53.026234 18027 leader_election.cc:302] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:53.026404 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:53.016604 19798 raft_consensus.cc:2726] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:53.026947 19479 raft_consensus.cc:2726] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:53.018784 19745 raft_consensus.cc:2726] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 03:38:53.021064 19492 consensus_queue.cc:260] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:53.036806 18249 leader_election.cc:302] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:53.056573 19492 raft_consensus.cc:3037] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:53.017402 19803 raft_consensus.cc:686] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:53.058272 18027 leader_election.cc:302] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:53.031972 19793 raft_consensus.cc:482] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
W20260807 03:38:53.032398 18223 debug-util.cc:398] Leaking SignalData structure 0x7b080034e920 after lost signal to thread 18346
I20260807 03:38:53.056277 19659 raft_consensus.cc:3037] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:53.031849 19544 raft_consensus.cc:729] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:53.093603 19544 consensus_queue.cc:260] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:53.095395 19544 raft_consensus.cc:3037] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:53.023794 19490 raft_consensus.cc:504] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:53.035084 19650 raft_consensus.cc:2726] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:53.098075 19490 leader_election.cc:288] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:53.036067 19745 raft_consensus.cc:2781] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:38:53.035602 19789 raft_consensus.cc:3032] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:53.091811 18180 raft_consensus.cc:1226] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20260807 03:38:53.092404 18223 debug-util.cc:398] Leaking SignalData structure 0x7b08002fb260 after lost signal to thread 19330
W20260807 03:38:53.114835 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 81798.6us
I20260807 03:38:53.030948 19612 consensus_queue.cc:1035] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.083s
I20260807 03:38:53.102275 18027 leader_election.cc:302] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:53.087755 19798 raft_consensus.cc:2726] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:53.116608 18141 leader_election.cc:302] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:53.123656 19747 consensus_queue.cc:1035] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 03:38:53.127264 19794 consensus_queue.cc:1046] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:53.117139 19745 raft_consensus.cc:686] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [term 2 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:53.107342 18298 raft_consensus.cc:1226] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:53.101320 19789 raft_consensus.cc:729] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:53.143843 18141 leader_election.cc:302] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
W20260807 03:38:53.115607 18223 debug-util.cc:398] Leaking SignalData structure 0x7b08003197a0 after lost signal to thread 19453
I20260807 03:38:53.145641 19803 consensus_queue.cc:237] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:53.131698 18027 leader_election.cc:302] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:53.139328 19794 raft_consensus.cc:3032] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:53.151381 19794 raft_consensus.cc:729] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:53.153144 18183 raft_consensus.cc:1226] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:53.152593 19794 consensus_queue.cc:260] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:53.153791 19745 consensus_queue.cc:237] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:53.154625 19794 raft_consensus.cc:3037] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:53.118821 19793 raft_consensus.cc:504] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:53.132962 19705 raft_consensus.cc:2726] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:38:53.161440 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:53.149889 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:53.164340 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:53.164919 18309 raft_consensus.cc:2454] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:38:53.035645 19773 consensus_queue.cc:260] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:53.168656 19773 raft_consensus.cc:3037] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:53.177564 18082 raft_consensus.cc:1261] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260807 03:38:53.179556 18082 pending_rounds.cc:76] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d: Aborting all ops after (but not including) 1
I20260807 03:38:53.176362 18141 leader_election.cc:302] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:53.179363 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:53.148717 19744 raft_consensus.cc:2781] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260807 03:38:53.155612 18223 debug-util.cc:398] Leaking SignalData structure 0x7b08003193c0 after lost signal to thread 19483
I20260807 03:38:53.188055 18188 raft_consensus.cc:1261] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:38:53.179812 18082 pending_rounds.cc:97] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260807 03:38:53.188760 18082 raft_consensus.cc:2866] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 03:38:53.168467 19783 consensus_queue.cc:1046] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:53.193574 18141 leader_election.cc:302] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:53.117220 19728 raft_consensus.cc:504] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:53.163203 19594 raft_consensus.cc:2781] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:53.204541 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:53.178097 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:53.180941 19604 raft_consensus.cc:3037] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:53.207969 18190 raft_consensus.cc:2454] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 2.
I20260807 03:38:53.198046 19594 raft_consensus.cc:482] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:53.209810 18069 raft_consensus.cc:1226] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 1. Current term is 2. Ops: []
W20260807 03:38:53.195497 18223 debug-util.cc:398] Leaking SignalData structure 0x7b0800319480 after lost signal to thread 19548
I20260807 03:38:53.186707 19744 raft_consensus.cc:482] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:53.194479 19750 raft_consensus.cc:2726] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 03:38:53.192972 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:53.205631 19769 raft_consensus.cc:2726] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 03:38:53.209826 19728 leader_election.cc:288] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:53.221038 19594 raft_consensus.cc:3037] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
W20260807 03:38:53.236490 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 46132us
W20260807 03:38:53.225201 18223 debug-util.cc:398] Leaking SignalData structure 0x7b0800319520 after lost signal to thread 19582
I20260807 03:38:53.158789 19797 consensus_queue.cc:1046] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.075s
I20260807 03:38:53.224730 19744 raft_consensus.cc:3037] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:53.247512 18027 leader_election.cc:302] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:53.261989 18248 leader_election.cc:302] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:53.263888 18027 leader_election.cc:302] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:53.268438 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:53.269184 18309 raft_consensus.cc:3037] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:53.216970 19768 consensus_queue.cc:1035] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20260807 03:38:53.269692 18072 raft_consensus.cc:2454] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
I20260807 03:38:53.272455 18027 leader_election.cc:302] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
W20260807 03:38:53.261492 18223 debug-util.cc:398] Leaking SignalData structure 0x7b08003250e0 after lost signal to thread 19609
I20260807 03:38:53.245970 19819 raft_consensus.cc:482] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:53.277338 18079 raft_consensus.cc:1226] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:53.281678 19726 raft_consensus.cc:2781] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260807 03:38:53.283077 19807 log.cc:924] Time spent T 00000000000000000000000000000000 P 8da2065a77024f35bdfc0dc3adb83f69: Append to log took a long time: real 0.175s user 0.000s sys 0.004s
I20260807 03:38:53.196508 19789 consensus_queue.cc:260] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:53.296633 18248 leader_election.cc:302] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:38:53.264477 19479 raft_consensus.cc:2726] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 03:38:53.300725 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
W20260807 03:38:53.276940 19813 log.cc:924] Time spent T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.058s user 0.001s sys 0.000s
W20260807 03:38:53.278750 18223 debug-util.cc:398] Leaking SignalData structure 0x7b08003193e0 after lost signal to thread 19615
W20260807 03:38:53.298827 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 69662.3us
W20260807 03:38:53.304797 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 92721.6us
I20260807 03:38:53.305892 18309 raft_consensus.cc:2454] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 2.
I20260807 03:38:53.366154 18190 raft_consensus.cc:1226] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:53.374074 18141 leader_election.cc:302] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:53.294169 19726 raft_consensus.cc:482] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
W20260807 03:38:53.308297 19645 consensus_meta.cc:287] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 0.530s user 0.010s sys 0.000s
I20260807 03:38:53.380232 19787 raft_consensus.cc:2726] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:53.363067 19495 raft_consensus.cc:2726] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 03:38:53.366597 19625 raft_consensus.cc:482] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:53.396397 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:53.384869 19789 raft_consensus.cc:3037] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:53.402132 18141 leader_election.cc:302] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:53.404318 19583 raft_consensus.cc:2726] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 03:38:53.382241 19726 raft_consensus.cc:3037] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:53.421080 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
W20260807 03:38:53.421533 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 124980us
I20260807 03:38:53.392549 19793 leader_election.cc:288] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:53.376662 19797 consensus_queue.cc:1046] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
W20260807 03:38:53.431872 19743 consensus_meta.cc:287] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.511s user 0.009s sys 0.000s
W20260807 03:38:53.449348 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 143148us
I20260807 03:38:53.395041 19625 raft_consensus.cc:504] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:53.452633 18311 raft_consensus.cc:2454] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 2.
I20260807 03:38:53.453066 18027 leader_election.cc:302] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:53.297497 19819 raft_consensus.cc:504] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:38:53.462491 18300 raft_consensus.cc:1226] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:53.467697 18311 raft_consensus.cc:1226] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:53.450454 19743 raft_consensus.cc:2726] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:53.473131 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:53.472244 18027 leader_election.cc:302] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:53.475164 18027 leader_election.cc:302] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:53.477150 18027 leader_election.cc:302] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
W20260807 03:38:53.482156 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 49064.9us
I20260807 03:38:53.482282 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:53.457221 19578 raft_consensus.cc:2726] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:38:53.482517 19705 raft_consensus.cc:2781] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:38:53.479121 19645 raft_consensus.cc:2726] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:53.482015 19650 raft_consensus.cc:2781] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:38:53.460691 19492 consensus_queue.cc:1046] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20260807 03:38:53.455415 19641 consensus_queue.cc:1035] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20260807 03:38:53.467005 19797 consensus_queue.cc:1046] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
W20260807 03:38:53.538161 19818 log.cc:924] Time spent T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.076s user 0.000s sys 0.001s
W20260807 03:38:53.556169 19814 log.cc:924] Time spent T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.195s user 0.001s sys 0.000s
I20260807 03:38:53.497215 19719 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:53.535550 19650 raft_consensus.cc:686] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:38:53.547885 19828 raft_consensus.cc:2781] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:38:53.548038 19604 raft_consensus.cc:2726] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:53.563338 19704 raft_consensus.cc:482] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:53.554342 19801 raft_consensus.cc:2726] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 03:38:53.573383 18309 raft_consensus.cc:1226] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 3. Current term is 4. Ops: []
I20260807 03:38:53.572618 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:53.572863 19828 raft_consensus.cc:686] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 2 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:53.585021 19659 raft_consensus.cc:2726] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:53.591015 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:53.579907 18185 raft_consensus.cc:1226] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:53.601253 19819 leader_election.cc:288] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:53.576853 19828 consensus_queue.cc:237] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:53.599292 19625 leader_election.cc:288] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:53.609732 18191 raft_consensus.cc:1226] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 3. Current term is 4. Ops: []
I20260807 03:38:53.596632 19594 raft_consensus.cc:504] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:53.653319 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:53.639442 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:53.657801 18183 raft_consensus.cc:2454] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 2.
I20260807 03:38:53.660133 19496 raft_consensus.cc:3032] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:53.660627 19496 raft_consensus.cc:729] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
W20260807 03:38:53.172387 19488 consensus_queue.cc:1159] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "3c4bceb836db4536a0f255fdf86abc02" responder_term: 3 status { last_received { term: 1 index: 1 } last_committed_idx: 1 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 2. Current term is 3. Ops: [2.2-2.2]" } } last_received_current_leader { term: 0 index: 0 } }
I20260807 03:38:53.664247 19719 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:53.664402 19594 leader_election.cc:288] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:38:53.642490 19544 consensus_meta.cc:287] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 0.547s user 0.009s sys 0.003s
I20260807 03:38:53.661702 19496 consensus_queue.cc:260] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:53.563953 19704 raft_consensus.cc:504] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:53.613255 19650 consensus_queue.cc:237] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:53.647548 19744 raft_consensus.cc:504] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:53.725930 18190 raft_consensus.cc:1226] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 1. Current term is 2. Ops: []
W20260807 03:38:53.726752 19794 consensus_meta.cc:287] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 0.572s user 0.012s sys 0.001s
W20260807 03:38:53.823098 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 350513us
W20260807 03:38:53.809098 19773 consensus_meta.cc:287] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.552s user 0.008s sys 0.003s
W20260807 03:38:53.839200 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 361040us
I20260807 03:38:53.839949 18027 leader_election.cc:302] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:38:53.837592 19625 raft_consensus.cc:482] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:53.874814 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
W20260807 03:38:53.876168 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 377840us
I20260807 03:38:53.871536 18298 raft_consensus.cc:1261] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260807 03:38:53.869395 19705 raft_consensus.cc:686] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:38:53.893821 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:53.922843 19851 consensus_queue.cc:1046] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.024s
I20260807 03:38:53.924074 19851 raft_consensus.cc:482] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:53.897044 18298 pending_rounds.cc:76] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49: Aborting all ops after (but not including) 1
I20260807 03:38:53.929975 19854 consensus_queue.cc:1046] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20260807 03:38:53.931484 19854 raft_consensus.cc:3032] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:53.931910 19854 raft_consensus.cc:729] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:38:53.932942 19854 consensus_queue.cc:260] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:53.934569 19854 raft_consensus.cc:3037] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:53.936877 18298 pending_rounds.cc:97] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260807 03:38:53.875530 19625 raft_consensus.cc:504] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:53.937340 18298 raft_consensus.cc:2866] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 03:38:53.908993 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:53.928771 19650 raft_consensus.cc:3032] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:38:53.941689 18300 raft_consensus.cc:1226] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 2 LEADER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:53.952205 18295 raft_consensus.cc:1226] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 1. Current term is 2. Ops: []
W20260807 03:38:53.945457 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 60923.8us
I20260807 03:38:53.968667 18293 raft_consensus.cc:1226] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:53.969141 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:53.969709 18295 raft_consensus.cc:3037] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:53.928704 19793 raft_consensus.cc:3032] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Stepping down as leader of term 1
W20260807 03:38:53.977380 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 151773us
I20260807 03:38:53.937886 19704 leader_election.cc:288] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:53.939714 19744 leader_election.cc:288] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:53.939749 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:53.937088 19851 raft_consensus.cc:504] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:53.950301 19650 raft_consensus.cc:729] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:38:53.977043 19479 raft_consensus.cc:482] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 03:38:53.992466 19855 log.cc:924] Time spent T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.158s user 0.001s sys 0.000s
I20260807 03:38:53.995806 19583 raft_consensus.cc:3032] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:53.996286 19583 raft_consensus.cc:729] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:53.998083 18295 raft_consensus.cc:2454] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:38:53.978364 19793 raft_consensus.cc:729] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:53.998939 18293 raft_consensus.cc:1226] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:54.003054 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:53.990805 19832 raft_consensus.cc:2781] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:38:54.006263 18141 leader_election.cc:302] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:54.007416 18141 leader_election.cc:302] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:54.009191 18070 raft_consensus.cc:1261] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:38:54.010764 18073 raft_consensus.cc:1261] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:38:53.990007 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:54.014330 19844 raft_consensus.cc:2726] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:54.009938 19583 consensus_queue.cc:260] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:54.010347 19604 raft_consensus.cc:2726] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:54.000968 19705 consensus_queue.cc:237] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:54.030848 18079 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33268 (request call id 662) took 1028 ms. Trace:
I20260807 03:38:54.033058 18079 rpcz_store.cc:276] 0807 03:38:53.002452 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:53.004370 (+ 1918us) service_pool.cc:227] Handling call
0807 03:38:53.277155 (+272785us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:38:53.286116 (+ 8961us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:38:54.030674 (+744558us) spinlock_profiling.cc:243] Waited 272 ms on lock 0x7b54003782d0. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bea09 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:38:54.030799 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":626724224}
I20260807 03:38:54.034461 19750 consensus_queue.cc:1046] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:38:54.024765 18069 raft_consensus.cc:2379] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 2: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:38:54.041846 18141 leader_election.cc:302] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:54.043048 19845 raft_consensus.cc:2781] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:38:54.043677 19845 raft_consensus.cc:686] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:54.044730 19845 consensus_queue.cc:237] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:54.040171 19769 raft_consensus.cc:3032] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:54.049242 19769 raft_consensus.cc:729] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:54.027878 19719 raft_consensus.cc:482] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:54.051713 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:54.053521 18141 leader_election.cc:302] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:54.002132 19479 raft_consensus.cc:504] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:54.056819 18311 raft_consensus.cc:2454] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:38:54.030642 19650 consensus_queue.cc:260] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:54.076393 19726 consensus_meta.cc:287] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 0.645s user 0.007s sys 0.004s
I20260807 03:38:54.076812 19726 raft_consensus.cc:504] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:54.077415 18305 raft_consensus.cc:2454] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 2.
I20260807 03:38:54.077781 18081 raft_consensus.cc:1226] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:54.079423 19851 leader_election.cc:288] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:54.080214 19799 raft_consensus.cc:482] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:54.060616 19795 raft_consensus.cc:2726] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:38:54.089576 19870 raft_consensus.cc:482] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:54.037184 19793 consensus_queue.cc:260] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:54.060535 19787 raft_consensus.cc:482] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:54.040990 19492 consensus_queue.cc:1046] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
I20260807 03:38:54.050150 19769 consensus_queue.cc:260] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:54.068485 19744 raft_consensus.cc:3032] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:38:54.098454 19742 consensus_queue.cc:1035] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:54.091434 19583 raft_consensus.cc:3037] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:54.087395 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:54.081017 19799 raft_consensus.cc:504] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:38:54.108428 19744 raft_consensus.cc:729] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:54.090240 19870 raft_consensus.cc:504] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:54.065840 19719 raft_consensus.cc:504] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:54.096580 19604 consensus_queue.cc:1046] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.041s
I20260807 03:38:54.137982 19854 raft_consensus.cc:482] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:54.158690 19809 raft_consensus.cc:3032] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:54.158811 19854 raft_consensus.cc:504] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:54.161339 19854 leader_election.cc:288] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:54.159381 19809 raft_consensus.cc:729] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:54.125331 19625 leader_election.cc:288] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:54.105634 19787 raft_consensus.cc:504] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:54.170914 19765 raft_consensus.cc:482] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:54.171362 19870 leader_election.cc:288] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:54.107053 19617 consensus_queue.cc:1035] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.058s
I20260807 03:38:54.172217 18141 leader_election.cc:302] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:54.174692 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:54.175125 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:54.188282 19765 raft_consensus.cc:504] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:54.189118 19604 raft_consensus.cc:2781] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:54.189504 19604 raft_consensus.cc:482] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:54.189851 19604 raft_consensus.cc:3037] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:54.189827 19625 consensus_queue.cc:1046] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 03:38:54.190351 19765 leader_election.cc:288] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:54.190739 19625 consensus_queue.cc:1225] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer 66a36e1ac8e945e893668e282d394c49 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260807 03:38:54.193245 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:54.196637 19650 raft_consensus.cc:3037] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:38:54.197790 19604 raft_consensus.cc:504] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:54.200541 19604 leader_election.cc:288] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:54.201547 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:54.212177 18297 raft_consensus.cc:2454] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 2.
I20260807 03:38:54.163637 19851 raft_consensus.cc:482] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:54.219429 19851 raft_consensus.cc:504] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:54.221794 19851 leader_election.cc:288] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:54.226332 19492 raft_consensus.cc:482] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:54.227083 19492 raft_consensus.cc:504] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:54.229151 19479 leader_election.cc:288] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:54.231097 19719 leader_election.cc:288] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:54.231881 19492 leader_election.cc:288] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:54.232044 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:54.233067 18300 raft_consensus.cc:2454] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:38:54.234443 18141 leader_election.cc:302] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:54.235628 19492 raft_consensus.cc:2781] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:54.237716 19492 raft_consensus.cc:482] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:54.238186 19492 raft_consensus.cc:3037] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:54.240000 19291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:38:54.240962 19291 raft_consensus.cc:2454] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
I20260807 03:38:54.238511 19799 leader_election.cc:288] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:38:54.248450 19852 raft_consensus.cc:482] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:54.255650 18309 raft_consensus.cc:1261] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 03:38:54.249110 19852 raft_consensus.cc:504] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:54.265774 19604 consensus_queue.cc:1035] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.002s
I20260807 03:38:54.162452 19809 consensus_queue.cc:260] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:54.271855 18003 auto_rebalancer.cc:663] Could not move replica: Not found: Couldn't find leader for tablet 9a59a83a64df475ea15dd572f84024bd
W20260807 03:38:54.274816 18003 auto_rebalancer.cc:264] scheduled replica move failed to complete: Not found: Couldn't find leader for tablet 9a59a83a64df475ea15dd572f84024bd
I20260807 03:38:54.274108 18298 raft_consensus.cc:1226] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 2 LEADER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:54.243228 19787 leader_election.cc:288] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:38:54.276216 18184 consensus_meta.cc:287] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 0.608s user 0.010s sys 0.002s
I20260807 03:38:54.277948 19714 consensus_queue.cc:1046] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 03:38:54.278973 19714 consensus_queue.cc:1225] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer 66a36e1ac8e945e893668e282d394c49 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260807 03:38:54.278407 19852 leader_election.cc:288] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:54.280891 19787 raft_consensus.cc:482] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:54.281466 19787 raft_consensus.cc:504] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:54.244311 19744 consensus_queue.cc:260] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:54.295094 17915 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35098 (request call id 19) took 1837 ms. Trace:
I20260807 03:38:54.283840 19787 leader_election.cc:288] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:54.301219 17915 rpcz_store.cc:276] 0807 03:38:52.457621 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:52.457741 (+ 120us) service_pool.cc:227] Handling call
0807 03:38:54.295015 (+1837274us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 03:38:52.798783 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 03:38:52.860065 (+ 61282us) write_op.cc:432] Acquiring schema lock in shared mode
0807 03:38:52.862687 (+ 2622us) write_op.cc:435] Acquired schema lock
0807 03:38:52.862707 (+ 20us) tablet.cc:660] Decoding operations
0807 03:38:52.899170 (+ 36463us) write_op.cc:620] Acquiring the partition lock for write op
0807 03:38:52.899219 (+ 49us) write_op.cc:641] Partition lock acquired for write op
0807 03:38:52.899248 (+ 29us) tablet.cc:683] Acquiring locks for 3 operations
0807 03:38:52.899593 (+ 345us) tablet.cc:699] Row locks acquired
0807 03:38:52.899619 (+ 26us) write_op.cc:260] PREPARE: finished
0807 03:38:52.899774 (+ 155us) write_op.cc:270] Start()
0807 03:38:52.899876 (+ 102us) write_op.cc:276] Timestamp: P: 1786073932899736 usec, L: 0
0807 03:38:52.899896 (+ 20us) op_driver.cc:331] REPLICATION: starting
0807 03:38:52.900486 (+ 590us) log.cc:842] Serialized 1468 byte log entry
0807 03:38:53.232210 (+331724us) spinlock_profiling.cc:243] Waited 1.86 ms on lock 0x7b44000232cc. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f1a9a95 00007fe98f1aa4b2 00007fe991e52511 00007fe991e4de1a 00007fe991e41bee 00007fe991e40f36 00007fe991e40b58 00007fe991e4a551 00007fe991e4a509 00007fe991e4a499
0807 03:38:53.615697 (+383487us) op_driver.cc:444] REPLICATION: finished
0807 03:38:53.905098 (+289401us) write_op.cc:301] APPLY: starting
0807 03:38:53.905196 (+ 98us) tablet.cc:1365] starting BulkCheckPresence
0807 03:38:53.905364 (+ 168us) tablet.cc:1368] finished BulkCheckPresence
0807 03:38:53.905400 (+ 36us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 03:38:53.906861 (+ 1461us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 03:38:53.907090 (+ 229us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0807 03:38:53.907202 (+ 112us) write_op.cc:312] APPLY: finished
0807 03:38:53.907542 (+ 340us) log.cc:842] Serialized 34 byte log entry
0807 03:38:53.928874 (+ 21332us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:38:53.940047 (+ 11173us) write_op.cc:454] Released schema lock
0807 03:38:53.941559 (+ 1512us) write_op.cc:341] FINISH: Updating metrics
0807 03:38:53.961313 (+ 19754us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:38:53.962269 (+ 956us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":11709,"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":270942,"apply.run_cpu_time_us":2879,"apply.run_wall_time_us":58551,"num_ops":3,"prepare.queue_time_us":234535,"prepare.run_cpu_time_us":68429,"prepare.run_wall_time_us":201252,"raft.queue_time_us":160439,"raft.run_cpu_time_us":3303,"raft.run_wall_time_us":309114,"replication_time_us":708815,"spinlock_wait_cycles":4276608,"thread_start_us":556797,"threads_started":4,"wal-append.queue_time_us":138560}]]}
I20260807 03:38:54.292256 19726 leader_election.cc:288] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:38:54.315346 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 338787us
I20260807 03:38:54.277477 18184 raft_consensus.cc:2454] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 2.
I20260807 03:38:54.326330 18293 raft_consensus.cc:2454] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:38:54.335237 18141 leader_election.cc:302] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:54.310825 19832 raft_consensus.cc:686] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:38:54.338528 18190 raft_consensus.cc:1226] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:54.342541 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:54.342694 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:54.343060 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:54.343787 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:54.344394 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:54.344941 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:54.346175 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:54.346853 18188 raft_consensus.cc:2454] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 2.
I20260807 03:38:54.347007 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:54.347575 18190 raft_consensus.cc:2454] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3.
I20260807 03:38:54.326243 19726 consensus_queue.cc:1035] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20260807 03:38:54.349174 19851 raft_consensus.cc:482] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:54.355221 18190 raft_consensus.cc:1261] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:38:54.412496 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:54.411345 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:54.439507 18070 raft_consensus.cc:2454] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:38:54.413547 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:54.358777 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:38:54.402318 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 46043.1us
I20260807 03:38:54.358218 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:54.419394 18075 raft_consensus.cc:1261] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 03:38:54.441589 18027 leader_election.cc:302] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:54.441745 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:54.349900 19641 raft_consensus.cc:482] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:54.410199 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:54.397770 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:54.448724 19456 raft_consensus.cc:482] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:54.443027 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:54.449450 19456 raft_consensus.cc:504] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:38:54.459044 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:54.459771 18185 raft_consensus.cc:2454] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 2.
I20260807 03:38:54.444736 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:54.439769 18082 raft_consensus.cc:2379] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 2: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:38:54.466656 18188 raft_consensus.cc:2454] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
I20260807 03:38:54.352298 19844 raft_consensus.cc:482] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:54.449302 18189 raft_consensus.cc:2454] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 2.
I20260807 03:38:54.450119 19496 raft_consensus.cc:3037] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:54.461604 19803 raft_consensus.cc:3032] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:54.406744 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:54.414227 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:54.487814 19645 consensus_queue.cc:1046] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20260807 03:38:54.499687 19645 raft_consensus.cc:482] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:54.464073 19641 raft_consensus.cc:504] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:54.443281 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:54.500202 19645 raft_consensus.cc:504] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:54.501899 18310 raft_consensus.cc:2454] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 2.
I20260807 03:38:54.502322 19645 leader_election.cc:288] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:54.441138 18187 raft_consensus.cc:2454] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 2.
I20260807 03:38:54.477972 18082 raft_consensus.cc:1226] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:54.351444 19240 leader_election.cc:302] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:54.468740 18249 leader_election.cc:302] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:54.481624 18027 leader_election.cc:302] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:54.504635 18027 leader_election.cc:302] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:54.347965 19787 consensus_queue.cc:1046] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.128s
I20260807 03:38:54.505319 19743 raft_consensus.cc:2781] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:54.505753 19743 raft_consensus.cc:482] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:54.505877 19787 raft_consensus.cc:2781] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:54.506240 18027 leader_election.cc:302] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:54.506547 19787 raft_consensus.cc:482] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:54.487988 19803 raft_consensus.cc:729] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:54.507921 19594 raft_consensus.cc:2781] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:38:54.508334 19803 consensus_queue.cc:260] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:54.510054 19803 raft_consensus.cc:3037] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:54.510185 19870 raft_consensus.cc:2781] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:54.461422 19456 leader_election.cc:288] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:38:54.510704 19870 raft_consensus.cc:482] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:54.511101 19870 raft_consensus.cc:3037] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:54.515009 18310 raft_consensus.cc:1261] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:38:54.506114 19743 raft_consensus.cc:3037] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:54.468394 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:54.507043 19871 consensus_queue.cc:1046] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:54.475421 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:54.353955 19872 raft_consensus.cc:2781] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:38:54.407047 19801 raft_consensus.cc:482] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:54.423413 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:54.465715 19479 raft_consensus.cc:2726] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 03:38:54.461493 19744 raft_consensus.cc:3037] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:38:54.527277 19578 raft_consensus.cc:482] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:54.542584 19578 raft_consensus.cc:504] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:54.545838 19872 raft_consensus.cc:686] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:54.547086 19872 consensus_queue.cc:237] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:54.548552 19291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:38:54.508359 19594 raft_consensus.cc:482] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:54.507066 19787 raft_consensus.cc:3037] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:54.447705 19851 raft_consensus.cc:504] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:54.543267 18185 raft_consensus.cc:1261] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:38:54.447857 19719 consensus_queue.cc:1035] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.094s
I20260807 03:38:54.413303 19861 consensus_queue.cc:1046] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.106s
I20260807 03:38:54.529974 19832 consensus_queue.cc:237] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:54.484937 19844 raft_consensus.cc:504] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:54.581615 19801 raft_consensus.cc:504] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:54.665804 18081 raft_consensus.cc:1261] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260807 03:38:54.901682 18081 pending_rounds.cc:76] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d: Aborting all ops after (but not including) 1
I20260807 03:38:54.901440 18027 leader_election.cc:302] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:54.902253 18081 pending_rounds.cc:97] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260807 03:38:54.902658 18081 raft_consensus.cc:2866] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 03:38:54.630030 19594 raft_consensus.cc:3037] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:38:54.686954 18187 raft_consensus.cc:1261] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260807 03:38:54.907401 19884 raft_consensus.cc:2781] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260807 03:38:54.727586 18184 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44662 (request call id 472) took 1547 ms (client timeout 1511 ms). Trace:
W20260807 03:38:54.908599 18184 rpcz_store.cc:269] 0807 03:38:53.180251 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:53.181008 (+ 757us) service_pool.cc:227] Handling call
0807 03:38:53.561575 (+380567us) spinlock_profiling.cc:243] Waited 9.14 ms on lock 0x7b5800003998. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f25f851 00007fe98f25f702 00007fe99814fb09 00007fe99708b730 00007fe99715179b 00007fe997151821 00007fe9971011b0 00007fe997100a94 00007fe9907e8c04 00007fe9907e8b92
0807 03:38:54.727285 (+1165710us) spinlock_profiling.cc:243] Waited 91.5 ms on lock 0x7b5400012554. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:38:54.727450 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":231494784}
W20260807 03:38:54.909418 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 379659us
I20260807 03:38:54.749711 19544 raft_consensus.cc:3032] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:54.668756 19798 raft_consensus.cc:3032] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:54.911087 19544 raft_consensus.cc:729] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:54.911804 19798 raft_consensus.cc:729] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:38:54.912932 19544 consensus_queue.cc:260] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:54.912992 19798 consensus_queue.cc:260] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:54.652843 18072 raft_consensus.cc:2379] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 2: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:38:54.915948 19769 raft_consensus.cc:3037] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:54.917021 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:54.921350 19809 raft_consensus.cc:3037] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:54.922644 17914 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39492 (request call id 144) took 2012 ms. Trace:
I20260807 03:38:54.929093 18190 raft_consensus.cc:2454] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
I20260807 03:38:54.707422 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:54.968830 17914 rpcz_store.cc:276] 0807 03:38:52.910617 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:52.910775 (+ 158us) service_pool.cc:227] Handling call
0807 03:38:53.019545 (+108770us) spinlock_profiling.cc:243] Waited 78.2 ms on lock 0x7b54002611d4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144af0 00007fe9921c7f2a 00007fe998119d6c 00007fe998213c3e 00007fe996525fb4 00007fe996525f42 00007fe996525e71 00007fe996525dec 00007fe9965250a2 00007fe9905e994c 00007fe9905e8d86
0807 03:38:54.922580 (+1903035us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":13,"spinlock_wait_cycles":179921664}
I20260807 03:38:54.937417 19793 raft_consensus.cc:3037] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:54.736752 19705 raft_consensus.cc:3032] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:54.908020 19884 raft_consensus.cc:482] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:54.766733 19871 raft_consensus.cc:482] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:54.741806 19829 raft_consensus.cc:482] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
W20260807 03:38:54.952232 19492 consensus_meta.cc:287] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 0.713s user 0.011s sys 0.000s
I20260807 03:38:54.993407 18193 raft_consensus.cc:1261] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:38:54.991923 17920 catalog_manager.cc:5495] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 1 to 3, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 3 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
W20260807 03:38:54.733203 19240 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 397767us
I20260807 03:38:54.952026 19797 raft_consensus.cc:482] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:54.965008 19874 raft_consensus.cc:482] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:54.964285 19604 raft_consensus.cc:482] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:54.968026 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:54.765137 19743 raft_consensus.cc:504] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:38:54.994166 19705 raft_consensus.cc:729] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
W20260807 03:38:55.099375 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 188389us
I20260807 03:38:55.117980 19604 raft_consensus.cc:504] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:55.118877 19799 consensus_queue.cc:1035] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:55.119150 19797 raft_consensus.cc:504] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:55.119190 19874 raft_consensus.cc:504] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:55.017310 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 115979us
I20260807 03:38:55.132596 18027 leader_election.cc:302] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
W20260807 03:38:55.140759 19870 consensus_meta.cc:287] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 0.629s user 0.009s sys 0.003s
I20260807 03:38:55.141352 19870 raft_consensus.cc:504] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:55.143786 19870 leader_election.cc:288] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:55.152402 19870 raft_consensus.cc:482] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:55.152998 19870 raft_consensus.cc:504] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:55.155421 19870 leader_election.cc:288] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:55.076740 18185 raft_consensus.cc:2379] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 2: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:38:55.180177 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:38:55.103399 19886 log.cc:924] Time spent T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.139s user 0.000s sys 0.000s
I20260807 03:38:55.002969 19544 raft_consensus.cc:3037] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:54.996351 19884 raft_consensus.cc:3037] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:55.002422 18184 raft_consensus.cc:1261] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:38:55.081447 18069 raft_consensus.cc:1226] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:55.101009 19835 raft_consensus.cc:482] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:55.060276 19617 raft_consensus.cc:482] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:55.073287 19729 raft_consensus.cc:482] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:55.103152 19825 raft_consensus.cc:482] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:55.102734 19496 raft_consensus.cc:482] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:55.174476 19798 raft_consensus.cc:3037] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:55.178926 19471 raft_consensus.cc:482] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:55.171857 18305 raft_consensus.cc:2454] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 2.
I20260807 03:38:55.167790 19799 consensus_queue.cc:1035] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.052s
I20260807 03:38:55.206694 19769 raft_consensus.cc:482] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:55.083302 19787 raft_consensus.cc:504] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:55.218382 18183 raft_consensus.cc:1261] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:38:55.165267 19492 raft_consensus.cc:504] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:55.118806 19705 consensus_queue.cc:260] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:55.230774 19388 raft_consensus.cc:482] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:55.002264 19871 raft_consensus.cc:504] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:38:55.230973 19744 raft_consensus.cc:482] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:55.027901 19829 raft_consensus.cc:504] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:55.242483 19870 raft_consensus.cc:2726] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 03:38:55.221590 19835 raft_consensus.cc:504] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:55.321756 19743 leader_election.cc:288] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:38:55.322754 19835 leader_election.cc:288] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:55.262735 19765 raft_consensus.cc:482] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:55.323833 19797 leader_election.cc:288] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:55.324437 19797 raft_consensus.cc:482] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:55.324241 19388 raft_consensus.cc:504] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:55.324899 19797 raft_consensus.cc:504] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:55.239259 19471 raft_consensus.cc:504] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:55.326900 19797 leader_election.cc:288] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:55.257115 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:55.329792 19829 leader_election.cc:288] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:55.332144 19744 raft_consensus.cc:504] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:55.333684 18070 raft_consensus.cc:1261] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:38:55.334160 19744 leader_election.cc:288] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:55.327831 18072 raft_consensus.cc:2454] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
I20260807 03:38:55.323443 19765 raft_consensus.cc:504] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:55.339578 18079 raft_consensus.cc:1226] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 2. Current term is 3. Ops: []
I20260807 03:38:55.326942 19471 leader_election.cc:288] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:55.223249 19617 raft_consensus.cc:504] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:55.225914 19729 raft_consensus.cc:504] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:55.227766 19496 raft_consensus.cc:504] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:55.318416 19719 raft_consensus.cc:482] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:55.228463 19825 raft_consensus.cc:504] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:55.361424 18140 leader_election.cc:302] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:55.363301 18248 leader_election.cc:302] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:55.257761 19769 raft_consensus.cc:504] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:55.350142 19870 raft_consensus.cc:482] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:55.372233 18140 leader_election.cc:302] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:55.366211 19853 raft_consensus.cc:2726] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:55.365832 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:55.374341 19705 raft_consensus.cc:3037] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:55.371569 19797 raft_consensus.cc:482] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:55.366364 19835 consensus_queue.cc:1035] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20260807 03:38:55.412114 19871 leader_election.cc:288] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:38:55.412714 19769 leader_election.cc:288] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:55.375157 19870 raft_consensus.cc:504] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:55.413945 19594 consensus_meta.cc:287] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 0.510s user 0.009s sys 0.000s
I20260807 03:38:55.414326 19594 raft_consensus.cc:504] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:55.414841 19870 leader_election.cc:288] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:55.417203 19496 leader_election.cc:288] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:55.399181 19765 leader_election.cc:288] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:55.418965 19594 leader_election.cc:288] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:55.388177 19604 leader_election.cc:288] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:55.367376 19719 raft_consensus.cc:504] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:55.420518 19797 raft_consensus.cc:504] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:55.421607 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 289095us
I20260807 03:38:55.385064 19704 raft_consensus.cc:2726] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:38:55.427218 18027 leader_election.cc:302] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:55.433209 18027 leader_election.cc:302] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:55.375105 19861 consensus_queue.cc:1035] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.026s
I20260807 03:38:55.421432 19803 raft_consensus.cc:482] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:55.456889 18311 raft_consensus.cc:1261] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:38:55.421218 19719 leader_election.cc:288] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
W20260807 03:38:55.469728 19891 log.cc:924] Time spent T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.121s user 0.001s sys 0.000s
I20260807 03:38:55.471611 19578 leader_election.cc:288] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:55.432171 19496 raft_consensus.cc:482] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:55.472643 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:55.472862 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:55.472945 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:55.392572 19847 consensus_queue.cc:1046] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.056s
I20260807 03:38:55.441730 19871 raft_consensus.cc:2781] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:55.432348 19594 raft_consensus.cc:482] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:55.472548 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:55.483989 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:55.455998 19704 raft_consensus.cc:482] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:55.498697 18079 raft_consensus.cc:2454] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:38:55.499797 18081 raft_consensus.cc:3037] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:55.471719 19292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331"
I20260807 03:38:55.485091 19479 raft_consensus.cc:2781] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:55.477500 18072 raft_consensus.cc:1226] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:55.504527 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:55.493345 19492 leader_election.cc:288] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:55.555841 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:55.493218 19787 leader_election.cc:288] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:55.574499 18073 raft_consensus.cc:1261] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:38:55.587253 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:55.587623 18081 raft_consensus.cc:2454] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:38:55.587921 18075 raft_consensus.cc:2454] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:38:55.588521 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:38:55.589596 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 217474us
W20260807 03:38:55.582617 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 62647.4us
I20260807 03:38:55.593353 18070 raft_consensus.cc:1226] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:55.594154 18081 raft_consensus.cc:2354] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 2. Current term is 3.
I20260807 03:38:55.594610 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:55.457866 19897 consensus_queue.cc:1035] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
I20260807 03:38:55.492789 19578 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:55.507433 19871 raft_consensus.cc:482] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:55.586004 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:55.510399 19583 raft_consensus.cc:482] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:55.606191 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:55.485240 19853 consensus_queue.cc:1035] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.099s
I20260807 03:38:55.551098 19902 raft_consensus.cc:482] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:55.552981 19797 leader_election.cc:288] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:55.515722 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:55.612151 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:55.583401 19479 raft_consensus.cc:482] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:55.577764 19291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:38:55.570245 19617 leader_election.cc:288] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:55.570493 19825 leader_election.cc:288] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:55.621084 18141 leader_election.cc:302] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:55.555570 19594 raft_consensus.cc:504] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:55.550390 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:55.500459 19496 raft_consensus.cc:504] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:55.578231 19832 raft_consensus.cc:482] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:55.479952 19803 raft_consensus.cc:504] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:55.637943 18080 raft_consensus.cc:1226] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 3. Current term is 4. Ops: []
I20260807 03:38:55.570241 18297 raft_consensus.cc:2454] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 2.
I20260807 03:38:55.577980 19892 raft_consensus.cc:482] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:55.638765 18027 leader_election.cc:302] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:55.592991 19729 leader_election.cc:288] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:55.610872 19829 raft_consensus.cc:2781] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:55.676566 19892 raft_consensus.cc:504] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:55.644810 19479 raft_consensus.cc:3037] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:55.677497 19844 leader_election.cc:288] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:55.678524 19892 leader_election.cc:288] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:55.679023 19801 leader_election.cc:288] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:55.679697 19852 raft_consensus.cc:3032] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:55.680064 19852 raft_consensus.cc:729] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:55.680863 19852 consensus_queue.cc:260] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:55.681958 19803 leader_election.cc:288] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:55.683346 19388 leader_election.cc:288] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:55.683964 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:55.684677 19479 raft_consensus.cc:504] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:55.686317 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:55.686851 19479 leader_election.cc:288] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:55.686986 18184 raft_consensus.cc:2454] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 4.
I20260807 03:38:55.688612 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:55.680881 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:55.689455 18184 raft_consensus.cc:3037] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:55.689770 18191 raft_consensus.cc:2454] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 2.
I20260807 03:38:55.643275 19902 raft_consensus.cc:504] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:55.615177 19871 raft_consensus.cc:3037] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:55.609202 19728 raft_consensus.cc:2726] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:38:55.611469 19847 raft_consensus.cc:482] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:55.558239 19704 raft_consensus.cc:504] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:55.702678 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:55.623869 19583 raft_consensus.cc:504] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:55.703857 19704 leader_election.cc:288] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:55.622786 19765 raft_consensus.cc:482] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:55.704107 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:55.704571 19765 raft_consensus.cc:504] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:55.705355 19583 leader_election.cc:288] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:55.706362 19765 leader_election.cc:288] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:38:55.656744 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 65528.8us
I20260807 03:38:55.707115 18248 leader_election.cc:302] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:55.668742 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:55.656566 19898 raft_consensus.cc:482] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:55.671672 19917 raft_consensus.cc:2726] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:55.672117 19641 leader_election.cc:288] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:55.692358 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:55.708186 19898 raft_consensus.cc:504] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:55.689605 19829 raft_consensus.cc:482] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:55.666316 18193 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44660 (request call id 838) took 1242 ms. Trace:
I20260807 03:38:55.709231 19829 raft_consensus.cc:3037] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:55.672770 18183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44660 (request call id 823) took 1318 ms. Trace:
I20260807 03:38:55.671758 18310 raft_consensus.cc:3037] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 3
I20260807 03:38:55.724506 19898 leader_election.cc:288] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:55.725113 19594 leader_election.cc:288] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:38:55.693410 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 54759.2us
I20260807 03:38:55.693595 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:55.612572 19578 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:55.700556 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:55.701579 18140 leader_election.cc:302] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:55.667853 19832 raft_consensus.cc:504] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:55.673429 19496 leader_election.cc:288] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:55.708154 18309 raft_consensus.cc:2454] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 2.
I20260807 03:38:55.703850 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:55.709414 18193 rpcz_store.cc:276] 0807 03:38:54.423518 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:54.502636 (+ 79118us) service_pool.cc:227] Handling call
0807 03:38:54.502970 (+ 334us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:38:55.121316 (+618346us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:38:55.666063 (+544747us) spinlock_profiling.cc:243] Waited 490 ms on lock 0x7b5400253354. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:38:55.666249 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1127066368}
I20260807 03:38:55.714191 18075 raft_consensus.cc:2454] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:55.715217 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:55.716773 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:55.720387 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:55.721438 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:55.722165 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:55.725740 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:55.692294 19902 leader_election.cc:288] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:55.753754 18249 leader_election.cc:302] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:55.714799 18183 rpcz_store.cc:276] 0807 03:38:54.354304 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:54.455603 (+101299us) service_pool.cc:227] Handling call
0807 03:38:54.455905 (+ 302us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:38:55.258747 (+802842us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:38:55.672542 (+413795us) spinlock_profiling.cc:243] Waited 705 ms on lock 0x7b54001de054. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:38:55.672717 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1621099392}
I20260807 03:38:55.790092 18027 leader_election.cc:302] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:55.789901 19471 raft_consensus.cc:2781] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 03:38:55.790836 19471 raft_consensus.cc:686] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:55.791216 18069 raft_consensus.cc:2454] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:55.793605 19832 leader_election.cc:288] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:55.794150 19832 raft_consensus.cc:2781] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:55.795015 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:55.795630 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:38:55.752552 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 45516.3us
I20260807 03:38:55.796790 18187 raft_consensus.cc:2454] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
W20260807 03:38:55.797086 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 95586us
I20260807 03:38:55.797639 18140 leader_election.cc:302] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:55.797930 18073 raft_consensus.cc:2454] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
W20260807 03:38:55.725889 19884 consensus_meta.cc:287] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331: Time spent flushing consensus metadata: real 0.504s user 0.007s sys 0.002s
I20260807 03:38:55.798728 18081 raft_consensus.cc:2454] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:55.772428 19874 leader_election.cc:288] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:55.795682 18189 raft_consensus.cc:3037] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
W20260807 03:38:55.770515 19798 consensus_meta.cc:287] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 0.542s user 0.009s sys 0.000s
I20260807 03:38:55.806282 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:55.806983 18082 raft_consensus.cc:2454] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:38:55.753923 19835 raft_consensus.cc:482] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:55.807900 19835 raft_consensus.cc:504] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:55.808202 19914 consensus_queue.cc:1046] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:55.809983 19835 leader_election.cc:288] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:55.804620 19794 raft_consensus.cc:2781] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:55.810482 18248 leader_election.cc:398] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Vote denied by peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163) with higher term. Message: Invalid argument: T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 2. Current term is 3.
I20260807 03:38:55.804737 19544 raft_consensus.cc:2781] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:55.810837 18248 leader_election.cc:401] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260807 03:38:55.807056 18185 raft_consensus.cc:2454] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
I20260807 03:38:55.811036 19544 raft_consensus.cc:482] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:55.811504 19544 raft_consensus.cc:3037] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
W20260807 03:38:55.752447 19809 consensus_meta.cc:287] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 0.760s user 0.008s sys 0.002s
I20260807 03:38:55.819581 18027 leader_election.cc:302] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:55.794489 19832 raft_consensus.cc:482] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:55.814152 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:55.823580 18027 leader_election.cc:302] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:55.824029 18180 raft_consensus.cc:2379] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 4: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:38:55.816254 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:55.698977 19729 raft_consensus.cc:2726] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:55.752065 19871 raft_consensus.cc:504] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:55.752710 19719 raft_consensus.cc:2726] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:38:55.826753 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:55.754032 19853 raft_consensus.cc:482] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:55.824565 18188 raft_consensus.cc:2454] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
I20260807 03:38:55.750676 19851 leader_election.cc:288] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:55.777036 19852 raft_consensus.cc:3037] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:55.791640 19471 consensus_queue.cc:237] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:55.810561 19794 raft_consensus.cc:482] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:55.832201 19726 raft_consensus.cc:2781] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:55.835317 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:55.837900 18140 leader_election.cc:302] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:55.829298 19594 raft_consensus.cc:2781] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:38:55.736030 18310 raft_consensus.cc:2454] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3.
I20260807 03:38:55.828578 19832 raft_consensus.cc:3037] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:55.826916 19797 consensus_queue.cc:1035] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20260807 03:38:55.841203 18072 raft_consensus.cc:2379] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 2: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:38:55.834746 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:55.866410 18027 leader_election.cc:302] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:55.849625 19853 raft_consensus.cc:504] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:55.834576 19872 raft_consensus.cc:482] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:55.868865 18187 raft_consensus.cc:2454] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
I20260807 03:38:55.858569 17912 catalog_manager.cc:5495] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 1 to 2, leader changed from d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 2 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:38:55.730165 19847 raft_consensus.cc:504] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:55.872298 18140 leader_election.cc:302] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:55.823278 19902 consensus_queue.cc:1035] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
W20260807 03:38:55.918929 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 52622us
I20260807 03:38:55.912783 19496 raft_consensus.cc:2726] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163) with higher term. Message: Invalid argument: T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 2. Current term is 3.
I20260807 03:38:55.868429 19726 raft_consensus.cc:482] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:55.859615 19860 raft_consensus.cc:482] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:55.930466 18027 leader_election.cc:302] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:55.812467 19884 raft_consensus.cc:504] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:38:55.868809 19794 raft_consensus.cc:3037] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:55.874351 19594 raft_consensus.cc:482] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
W20260807 03:38:55.926395 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 91135.4us
I20260807 03:38:55.912889 19861 raft_consensus.cc:2781] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:55.951311 19861 raft_consensus.cc:482] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:55.951712 19861 raft_consensus.cc:3037] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
W20260807 03:38:55.929515 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 57325.9us
I20260807 03:38:55.859366 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:55.927495 17912 catalog_manager.cc:5495] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 1 to 4, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 4 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:38:55.924144 19851 raft_consensus.cc:2726] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:55.930995 19726 raft_consensus.cc:3037] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:55.939620 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:55.940164 19594 raft_consensus.cc:3037] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:38:55.941150 18309 raft_consensus.cc:1261] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:38:55.944814 17920 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39494 (request call id 133) took 2820 ms. Trace:
I20260807 03:38:55.948737 18189 raft_consensus.cc:2454] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:38:55.915122 19728 consensus_queue.cc:1035] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20260807 03:38:55.949661 19578 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:55.953398 18140 leader_election.cc:302] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:55.923786 19798 raft_consensus.cc:482] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:55.965830 17912 catalog_manager.cc:5495] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 1 to 4, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 4 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:38:55.968713 18248 leader_election.cc:302] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:55.934919 19743 raft_consensus.cc:3032] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 03:38:55.935628 19874 raft_consensus.cc:2781] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:55.932917 19765 raft_consensus.cc:2781] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260807 03:38:56.001122 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 47799.1us
I20260807 03:38:55.964629 19388 raft_consensus.cc:482] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:55.953742 19929 raft_consensus.cc:482] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:56.033102 17912 catalog_manager.cc:5495] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 2 to 3, leader changed from d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 3 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:38:56.039357 18140 leader_election.cc:302] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:56.036828 19874 raft_consensus.cc:482] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:56.002516 17920 rpcz_store.cc:276] 0807 03:38:53.124699 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:53.126288 (+ 1589us) service_pool.cc:227] Handling call
0807 03:38:55.944755 (+2818467us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 03:38:55.550211 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 03:38:55.574752 (+ 24541us) write_op.cc:432] Acquiring schema lock in shared mode
0807 03:38:55.574793 (+ 41us) write_op.cc:435] Acquired schema lock
0807 03:38:55.574809 (+ 16us) tablet.cc:660] Decoding operations
0807 03:38:55.575112 (+ 303us) write_op.cc:620] Acquiring the partition lock for write op
0807 03:38:55.575313 (+ 201us) write_op.cc:641] Partition lock acquired for write op
0807 03:38:55.575340 (+ 27us) tablet.cc:683] Acquiring locks for 1 operations
0807 03:38:55.575583 (+ 243us) tablet.cc:699] Row locks acquired
0807 03:38:55.575603 (+ 20us) write_op.cc:260] PREPARE: finished
0807 03:38:55.575737 (+ 134us) write_op.cc:270] Start()
0807 03:38:55.575848 (+ 111us) write_op.cc:276] Timestamp: P: 1786073935575700 usec, L: 0
0807 03:38:55.575870 (+ 22us) op_driver.cc:331] REPLICATION: starting
0807 03:38:55.576555 (+ 685us) log.cc:842] Serialized 543 byte log entry
0807 03:38:55.717234 (+140679us) op_driver.cc:444] REPLICATION: finished
0807 03:38:55.719165 (+ 1931us) write_op.cc:301] APPLY: starting
0807 03:38:55.719248 (+ 83us) tablet.cc:1365] starting BulkCheckPresence
0807 03:38:55.719337 (+ 89us) tablet.cc:1368] finished BulkCheckPresence
0807 03:38:55.719480 (+ 143us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 03:38:55.719663 (+ 183us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 03:38:55.719773 (+ 110us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0807 03:38:55.719799 (+ 26us) write_op.cc:312] APPLY: finished
0807 03:38:55.719930 (+ 131us) log.cc:842] Serialized 22 byte log entry
0807 03:38:55.720095 (+ 165us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:38:55.720191 (+ 96us) write_op.cc:454] Released schema lock
0807 03:38:55.720228 (+ 37us) write_op.cc:341] FINISH: Updating metrics
0807 03:38:55.720479 (+ 251us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:38:55.720512 (+ 33us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":12,"child_traces":[["op",{"apply.queue_time_us":1644,"apply.run_cpu_time_us":1593,"apply.run_wall_time_us":1593,"num_ops":1,"prepare.queue_time_us":313777,"prepare.run_cpu_time_us":6496,"prepare.run_wall_time_us":150079,"raft.queue_time_us":69867,"raft.run_cpu_time_us":1156,"raft.run_wall_time_us":2383,"replication_time_us":141257,"thread_start_us":332456,"threads_started":4,"wal-append.queue_time_us":58350}]]}
I20260807 03:38:55.935024 19860 raft_consensus.cc:504] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:55.937139 19829 raft_consensus.cc:504] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.001047 19922 raft_consensus.cc:482] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:55.928229 19872 raft_consensus.cc:504] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.040724 19292 raft_consensus.cc:2379] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 3: Already voted for candidate 5c9b5062bdfa46af9396f1a85025a331 in this term.
I20260807 03:38:56.008077 19625 raft_consensus.cc:2781] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:56.035734 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:56.033775 19743 raft_consensus.cc:729] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:56.032218 18297 raft_consensus.cc:2454] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:38:56.057145 19871 leader_election.cc:288] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:56.006457 19799 consensus_queue.cc:1046] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.054s
I20260807 03:38:56.038959 19765 raft_consensus.cc:482] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:56.055662 18305 raft_consensus.cc:1226] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:56.034560 19798 raft_consensus.cc:504] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:38:56.047489 19388 raft_consensus.cc:504] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.071295 19874 raft_consensus.cc:3037] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:56.142748 18027 leader_election.cc:302] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:56.124349 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:56.116772 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:56.145980 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:38:56.131122 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 162500us
W20260807 03:38:56.138550 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 55410.6us
I20260807 03:38:56.116787 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:56.071509 19809 raft_consensus.cc:2781] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:56.160382 18082 raft_consensus.cc:2454] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:38:56.160442 18027 leader_election.cc:302] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:56.162176 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:56.167495 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:56.168256 18293 raft_consensus.cc:2454] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:38:56.162379 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:56.161602 18249 leader_election.cc:302] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
W20260807 03:38:56.117326 19938 log.cc:924] Time spent T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.120s user 0.000s sys 0.000s
I20260807 03:38:56.174916 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:38:56.105417 19941 log.cc:924] Time spent T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.259s user 0.001s sys 0.000s
I20260807 03:38:56.082206 19659 raft_consensus.cc:482] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:56.136003 19794 raft_consensus.cc:504] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.125286 18183 raft_consensus.cc:2454] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
I20260807 03:38:56.101569 19924 raft_consensus.cc:2781] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:56.159972 18311 raft_consensus.cc:2454] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:38:56.185598 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:56.185743 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:56.092064 19578 raft_consensus.cc:482] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:56.090698 19625 raft_consensus.cc:482] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:56.120898 18189 raft_consensus.cc:1226] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 2. Current term is 3. Ops: []
I20260807 03:38:56.113363 19948 raft_consensus.cc:482] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:56.190090 18079 raft_consensus.cc:2454] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
I20260807 03:38:56.095562 19825 consensus_queue.cc:1035] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.099s
I20260807 03:38:56.185967 19479 raft_consensus.cc:2726] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:38:56.184530 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:56.100631 19944 raft_consensus.cc:482] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:56.100710 19851 raft_consensus.cc:482] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:56.159530 19765 raft_consensus.cc:3037] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:56.092866 19949 raft_consensus.cc:482] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:56.089895 18184 raft_consensus.cc:2454] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3.
I20260807 03:38:56.205595 19853 leader_election.cc:288] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:56.159126 19480 raft_consensus.cc:482] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:56.159212 19898 raft_consensus.cc:2726] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 03:38:56.164363 19728 raft_consensus.cc:482] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:56.171605 19809 raft_consensus.cc:482] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:56.164203 19583 raft_consensus.cc:482] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:56.164243 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:56.213613 18187 raft_consensus.cc:2354] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 2. Current term is 3.
I20260807 03:38:56.134765 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ca0223f47b46fd955704eba5259370" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:56.205523 19924 raft_consensus.cc:482] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:56.228437 19955 raft_consensus.cc:2781] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:56.228662 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:56.211547 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:56.057271 19929 raft_consensus.cc:504] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.159895 19951 consensus_queue.cc:1035] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.081s
I20260807 03:38:56.202296 19659 raft_consensus.cc:504] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.088598 19922 raft_consensus.cc:504] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.239943 19944 raft_consensus.cc:504] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.262815 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:56.263118 19922 leader_election.cc:288] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:56.234738 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:56.263468 18082 raft_consensus.cc:2454] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
I20260807 03:38:56.263938 18188 raft_consensus.cc:2454] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 2.
I20260807 03:38:56.264253 18185 raft_consensus.cc:1261] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:38:56.264810 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:56.265259 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:56.189000 19847 leader_election.cc:288] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:56.191987 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:56.266098 18185 raft_consensus.cc:2379] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5c9b5062bdfa46af9396f1a85025a331 in current term 3: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:38:56.245173 19949 raft_consensus.cc:504] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.205179 19625 raft_consensus.cc:3037] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:56.173775 19743 consensus_queue.cc:260] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.244436 19871 raft_consensus.cc:482] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:56.266294 18310 raft_consensus.cc:2454] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:38:56.208451 19578 raft_consensus.cc:504] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.194844 19604 raft_consensus.cc:482] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:56.267479 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:56.251825 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:56.256995 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:56.250442 18298 raft_consensus.cc:2454] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:56.254136 19594 raft_consensus.cc:504] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.258649 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:56.278445 19860 leader_election.cc:288] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:56.249809 19809 raft_consensus.cc:3037] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:56.252921 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:56.263969 19944 leader_election.cc:288] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:56.259411 19388 leader_election.cc:288] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:56.279235 19743 raft_consensus.cc:3037] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:56.282040 18293 raft_consensus.cc:2454] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:38:56.284736 18082 raft_consensus.cc:2379] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5c9b5062bdfa46af9396f1a85025a331 in current term 3: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
W20260807 03:38:56.256990 19943 log.cc:924] Time spent T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.189s user 0.000s sys 0.001s
I20260807 03:38:56.281239 18180 consensus_queue.cc:237] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:38:56.248149 19480 raft_consensus.cc:504] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:56.244151 19923 log.cc:924] Time spent T 00000000000000000000000000000000 P 8da2065a77024f35bdfc0dc3adb83f69: Append to log took a long time: real 0.060s user 0.001s sys 0.000s
I20260807 03:38:56.257639 19955 raft_consensus.cc:482] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:56.256912 19872 leader_election.cc:288] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:56.249027 19832 raft_consensus.cc:504] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.246047 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:56.252954 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:56.258416 19929 leader_election.cc:288] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:56.322652 19594 leader_election.cc:288] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:56.316130 19578 leader_election.cc:288] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:56.316084 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:56.330129 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:56.370339 18193 raft_consensus.cc:3037] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:56.370638 18187 raft_consensus.cc:2379] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 4: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:38:56.315389 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:56.356601 19955 raft_consensus.cc:3037] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:56.285501 19829 leader_election.cc:288] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:56.290079 19719 raft_consensus.cc:482] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:56.249655 19728 raft_consensus.cc:504] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.250119 19583 raft_consensus.cc:504] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.380329 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:56.381129 19719 raft_consensus.cc:504] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.381881 19583 leader_election.cc:288] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:56.286371 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:56.382754 18185 raft_consensus.cc:2454] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 4.
I20260807 03:38:56.383068 19719 leader_election.cc:288] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:56.364166 19239 leader_election.cc:302] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:56.230640 19948 raft_consensus.cc:504] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.386962 19955 raft_consensus.cc:504] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.388468 19948 leader_election.cc:288] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:56.253233 19924 raft_consensus.cc:3037] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:56.252985 19851 raft_consensus.cc:504] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.290584 19884 leader_election.cc:288] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:38:56.359112 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 197544us
I20260807 03:38:56.298969 19794 leader_election.cc:288] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:56.389541 19884 raft_consensus.cc:2726] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 03:38:56.362317 18140 leader_election.cc:302] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:56.390504 19851 leader_election.cc:288] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:56.391337 19829 raft_consensus.cc:2781] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:38:56.391708 19829 raft_consensus.cc:482] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:56.392022 19829 raft_consensus.cc:3037] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:38:56.361272 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:56.393747 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:56.394282 18082 raft_consensus.cc:3037] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:56.295915 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:56.297451 18248 leader_election.cc:302] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:56.257090 18297 raft_consensus.cc:2454] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:38:56.396628 18305 raft_consensus.cc:2454] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:38:56.316406 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:56.397946 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:56.399132 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:56.400159 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:56.401139 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:56.401856 19728 leader_election.cc:288] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:56.371348 18080 raft_consensus.cc:2379] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 2: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:38:56.403385 18082 raft_consensus.cc:2454] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:38:56.334867 18027 leader_election.cc:302] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:56.288198 19871 raft_consensus.cc:504] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:38:56.408532 19291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:38:56.409318 19729 raft_consensus.cc:2781] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:38:56.409938 19729 raft_consensus.cc:686] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:38:56.410454 19871 leader_election.cc:288] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:38:56.294961 19604 raft_consensus.cc:504] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.410951 19729 consensus_queue.cc:237] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.315706 19765 raft_consensus.cc:504] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.416486 18140 leader_election.cc:302] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
W20260807 03:38:56.404510 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 69760.7us
I20260807 03:38:56.432703 18027 leader_election.cc:302] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:56.440632 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:56.357240 19726 raft_consensus.cc:504] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.442018 18027 leader_election.cc:302] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
W20260807 03:38:56.443213 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 53547.8us
W20260807 03:38:56.443747 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 151139us
I20260807 03:38:56.444049 19860 raft_consensus.cc:2781] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:56.375288 19659 leader_election.cc:288] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:56.446276 18248 leader_election.cc:302] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:56.456199 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:56.457334 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:56.449487 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:56.451110 19955 leader_election.cc:288] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:56.446265 19728 raft_consensus.cc:2781] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:56.464010 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:56.446571 19583 raft_consensus.cc:482] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:56.467444 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:38:56.454154 19239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 93777us
I20260807 03:38:56.453682 19797 raft_consensus.cc:2781] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:38:56.507609 18193 raft_consensus.cc:2454] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3.
I20260807 03:38:56.457270 19871 raft_consensus.cc:2781] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:38:56.469537 18027 leader_election.cc:302] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:38:56.473127 19726 leader_election.cc:288] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:38:56.475611 19861 consensus_meta.cc:287] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 0.523s user 0.004s sys 0.004s
I20260807 03:38:56.475238 19874 raft_consensus.cc:504] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:56.476336 19946 log.cc:924] Time spent T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.234s user 0.001s sys 0.000s
I20260807 03:38:56.476625 18072 raft_consensus.cc:1226] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 4. Current term is 5. Ops: []
I20260807 03:38:56.477110 19949 leader_election.cc:288] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:56.484218 18183 raft_consensus.cc:2354] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 2. Current term is 3.
I20260807 03:38:56.484093 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:56.484748 19798 leader_election.cc:288] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:38:56.485086 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:56.486074 19765 leader_election.cc:288] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:56.486907 19480 leader_election.cc:288] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:56.487226 19872 consensus_queue.cc:1035] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:56.459267 19479 raft_consensus.cc:482] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:56.468549 19728 raft_consensus.cc:482] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:56.449731 19851 consensus_queue.cc:1035] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
I20260807 03:38:56.489362 19955 raft_consensus.cc:482] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:56.466213 19948 raft_consensus.cc:2781] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:56.493806 19853 raft_consensus.cc:482] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:56.496577 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:56.497809 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:56.449903 19860 raft_consensus.cc:482] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:56.499332 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:56.491456 19924 raft_consensus.cc:504] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.451975 19922 raft_consensus.cc:482] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:56.567646 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:56.514600 19871 raft_consensus.cc:482] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 03:38:56.540943 19852 consensus_meta.cc:287] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.685s user 0.003s sys 0.007s
I20260807 03:38:56.503644 18249 leader_election.cc:302] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:56.568296 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:56.524050 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:38:56.572899 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 69347us
I20260807 03:38:56.543830 19902 raft_consensus.cc:2726] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:56.573812 18249 leader_election.cc:302] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:56.502516 19583 raft_consensus.cc:504] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.562505 19728 raft_consensus.cc:3037] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:56.577405 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:56.549472 18310 raft_consensus.cc:2454] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:38:56.562928 19948 raft_consensus.cc:482] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:56.582741 18298 raft_consensus.cc:2379] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 3: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:38:56.541543 18073 raft_consensus.cc:1226] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260807 03:38:56.652993 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:56.654991 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:56.656476 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:38:56.565181 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 78837.8us
I20260807 03:38:56.662909 18141 leader_election.cc:302] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:56.519793 18082 raft_consensus.cc:1226] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 2. Current term is 3. Ops: []
I20260807 03:38:56.670421 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:56.671576 18082 raft_consensus.cc:3037] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 3
I20260807 03:38:56.668217 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:56.647349 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:56.675792 18080 raft_consensus.cc:3037] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:56.650074 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:56.495424 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:56.579321 18249 leader_election.cc:302] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:56.509315 18180 raft_consensus.cc:1261] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:38:56.574887 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:56.684553 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:56.685750 18305 raft_consensus.cc:2454] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 2.
I20260807 03:38:56.687178 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:56.590849 18311 raft_consensus.cc:2396] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 03:38:56.588244 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:56.584259 19291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
W20260807 03:38:56.591334 19292 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59906 (request call id 41) took 1203 ms (client timeout 1565 ms). Trace:
I20260807 03:38:56.562255 19479 raft_consensus.cc:504] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:56.587613 19544 consensus_meta.cc:287] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 0.758s user 0.004s sys 0.007s
W20260807 03:38:56.692065 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 112675us
W20260807 03:38:56.691922 19292 rpcz_store.cc:269] 0807 03:38:55.416149 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:55.441221 (+ 25072us) service_pool.cc:227] Handling call
0807 03:38:56.591107 (+1149886us) spinlock_profiling.cc:243] Waited 438 ms on lock 0x7b5400611554. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:38:56.591256 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1008064768}
I20260807 03:38:56.692801 19544 raft_consensus.cc:504] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.592586 19583 leader_election.cc:288] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:56.695314 18249 leader_election.cc:398] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057) with higher term. Message: Invalid argument: T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 2. Current term is 3.
I20260807 03:38:56.695613 18025 leader_election.cc:302] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:38:56.608198 19625 raft_consensus.cc:504] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.583909 19604 leader_election.cc:288] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:56.697758 19544 leader_election.cc:288] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:56.698304 18184 raft_consensus.cc:2354] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 2. Current term is 3.
I20260807 03:38:56.698611 19544 raft_consensus.cc:2726] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:38:56.657774 18310 raft_consensus.cc:2454] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3.
I20260807 03:38:56.690831 18312 raft_consensus.cc:2379] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5c9b5062bdfa46af9396f1a85025a331 in current term 3: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:38:56.700956 19625 leader_election.cc:288] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:56.609831 19728 raft_consensus.cc:504] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.694806 19479 leader_election.cc:288] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:56.701393 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:56.701604 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:56.702529 18312 raft_consensus.cc:2354] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 2. Current term is 3.
I20260807 03:38:56.702364 19479 raft_consensus.cc:482] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:56.703109 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:56.703593 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:56.703881 19728 leader_election.cc:288] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:56.704527 19479 raft_consensus.cc:504] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.706624 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:56.706962 19728 raft_consensus.cc:2726] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:56.707618 18072 raft_consensus.cc:3037] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 3
I20260807 03:38:56.706962 19659 consensus_queue.cc:1046] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.002s
I20260807 03:38:56.709059 19719 raft_consensus.cc:2781] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:38:56.709586 19479 leader_election.cc:288] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:56.711041 19659 consensus_queue.cc:1046] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:56.709643 19719 raft_consensus.cc:686] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:56.712191 19471 raft_consensus.cc:3032] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 03:38:56.713169 19471 raft_consensus.cc:729] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:56.713549 18141 leader_election.cc:302] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:56.713821 19719 consensus_queue.cc:237] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:38:56.714594 19583 raft_consensus.cc:482] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:56.715480 19583 raft_consensus.cc:504] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.718603 19583 leader_election.cc:288] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:56.704370 18312 raft_consensus.cc:2454] T af8877c3fbba4ab18765832f1390921f P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:38:56.722653 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:56.723308 19471 consensus_queue.cc:260] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.724040 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:56.697579 18249 leader_election.cc:302] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:56.725214 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:38:56.690457 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 179434us
I20260807 03:38:56.571159 19797 raft_consensus.cc:686] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 3 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:56.606556 19829 raft_consensus.cc:504] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.602254 19809 raft_consensus.cc:504] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.737161 19809 leader_election.cc:288] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:56.583523 19852 raft_consensus.cc:2781] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:56.738891 19852 raft_consensus.cc:482] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:56.739362 19852 raft_consensus.cc:3037] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:56.567116 19853 raft_consensus.cc:504] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.567236 19860 raft_consensus.cc:3037] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:56.746425 19853 leader_election.cc:288] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:56.747303 19825 raft_consensus.cc:2781] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:56.747767 19825 raft_consensus.cc:482] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:56.523382 19861 raft_consensus.cc:504] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.571921 19871 raft_consensus.cc:3037] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:38:56.748101 19825 raft_consensus.cc:3037] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:56.750171 19861 leader_election.cc:288] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:56.591899 19874 leader_election.cc:288] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:56.593252 19924 leader_election.cc:288] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:56.574286 19922 raft_consensus.cc:504] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.608176 19945 raft_consensus.cc:3032] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:38:56.752468 19945 raft_consensus.cc:729] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:56.754076 19945 consensus_queue.cc:260] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.588218 19948 raft_consensus.cc:3037] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:56.564530 19955 raft_consensus.cc:504] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.760531 18310 raft_consensus.cc:2454] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:38:56.761502 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:56.761782 19955 leader_election.cc:288] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:56.577493 19957 raft_consensus.cc:2726] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260807 03:38:56.584059 19964 log.cc:924] Time spent T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.085s user 0.000s sys 0.001s
I20260807 03:38:56.724680 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:56.573617 19832 leader_election.cc:288] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:56.764593 18312 raft_consensus.cc:3037] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
W20260807 03:38:56.742082 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 44723.9us
I20260807 03:38:56.768854 19957 consensus_queue.cc:1046] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 03:38:56.771327 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:56.771590 18297 raft_consensus.cc:2454] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:56.781454 18027 leader_election.cc:302] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:56.783699 18027 leader_election.cc:302] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:56.751572 18188 raft_consensus.cc:2354] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 2. Current term is 3.
I20260807 03:38:56.771517 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:56.773324 19743 raft_consensus.cc:3032] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 03:38:56.785512 19743 raft_consensus.cc:729] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:56.786558 19743 consensus_queue.cc:260] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.787269 19832 raft_consensus.cc:2726] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:56.788285 19743 raft_consensus.cc:3037] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:56.786520 18312 raft_consensus.cc:2454] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3.
I20260807 03:38:56.795333 18027 leader_election.cc:302] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:56.772492 18189 raft_consensus.cc:2454] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
I20260807 03:38:56.763758 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
W20260807 03:38:56.798571 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 73162.6us
I20260807 03:38:56.776134 19797 consensus_queue.cc:237] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.801855 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:56.777628 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:56.804651 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:56.734056 19829 leader_election.cc:288] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:56.805474 18193 raft_consensus.cc:2454] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
I20260807 03:38:56.805168 19471 raft_consensus.cc:3037] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:38:56.776646 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:56.774282 19922 leader_election.cc:288] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:56.807075 18080 raft_consensus.cc:2454] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:38:56.805557 19583 raft_consensus.cc:2726] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:56.808198 18082 raft_consensus.cc:2454] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:38:56.756001 19945 raft_consensus.cc:3037] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:38:56.819582 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:56.819880 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:56.824916 18189 raft_consensus.cc:2454] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
I20260807 03:38:56.819787 19902 raft_consensus.cc:2726] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:56.827296 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:56.819849 18141 leader_election.cc:302] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:56.843282 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:56.844676 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:56.845434 18298 raft_consensus.cc:2454] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 2.
I20260807 03:38:56.846536 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:56.845533 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:56.847239 18298 raft_consensus.cc:3037] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:56.848548 19874 raft_consensus.cc:2726] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:38:56.849290 18070 raft_consensus.cc:1226] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 3. Current term is 4. Ops: []
W20260807 03:38:56.854233 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 59088.3us
I20260807 03:38:56.855278 18027 leader_election.cc:302] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:56.856967 19902 raft_consensus.cc:2726] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:38:56.859508 18027 leader_election.cc:398] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Vote denied by peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021) with higher term. Message: Invalid argument: T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 2. Current term is 3.
I20260807 03:38:56.873723 18311 raft_consensus.cc:1261] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260807 03:38:56.908013 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 66030.6us
I20260807 03:38:56.859889 18027 leader_election.cc:401] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260807 03:38:56.876662 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:56.903672 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:56.869467 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:56.839619 19719 raft_consensus.cc:482] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:56.864702 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ca0223f47b46fd955704eba5259370" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:56.909801 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:38:56.937348 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 77960.3us
I20260807 03:38:56.938830 18309 raft_consensus.cc:2454] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 2.
I20260807 03:38:56.877689 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:56.866531 19583 raft_consensus.cc:2781] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:56.917143 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:56.914268 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:56.867997 19292 raft_consensus.cc:1261] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 03:38:56.862767 19871 raft_consensus.cc:504] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:56.915084 19962 log.cc:924] Time spent T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.442s user 0.006s sys 0.000s
I20260807 03:38:56.941927 18305 raft_consensus.cc:3037] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:56.900645 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:56.876195 19948 raft_consensus.cc:504] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.947016 19583 raft_consensus.cc:482] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:56.941700 19874 raft_consensus.cc:2726] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021) with higher term. Message: Invalid argument: T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 2. Current term is 3.
I20260807 03:38:56.913483 19860 raft_consensus.cc:504] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.972066 17912 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39510 (request call id 139) took 2905 ms. Trace:
I20260807 03:38:56.882297 19829 consensus_queue.cc:1046] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.076s
I20260807 03:38:56.972151 18248 leader_election.cc:302] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:56.984612 18075 raft_consensus.cc:2354] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 4. Current term is 5.
I20260807 03:38:56.953883 19719 raft_consensus.cc:504] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.942209 19825 raft_consensus.cc:504] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.986032 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:56.983870 18027 leader_election.cc:302] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:56.964637 19583 raft_consensus.cc:3037] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:56.941605 19852 raft_consensus.cc:504] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:56.992211 18188 raft_consensus.cc:2379] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 3: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:38:56.991818 18185 raft_consensus.cc:2379] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 3: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
W20260807 03:38:57.002677 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 52478.7us
W20260807 03:38:57.003726 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 62663.4us
I20260807 03:38:57.007473 18300 raft_consensus.cc:2354] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 2. Current term is 3.
I20260807 03:38:57.008039 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:57.008391 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:57.008636 19719 leader_election.cc:288] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:57.010214 19924 raft_consensus.cc:2781] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:38:57.010769 19924 raft_consensus.cc:686] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:57.011801 19924 consensus_queue.cc:237] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:57.013187 19825 leader_election.cc:288] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:56.998390 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:57.017524 18183 raft_consensus.cc:2379] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 3: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:38:57.018205 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:57.019351 18297 raft_consensus.cc:1226] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260807 03:38:57.008842 18187 raft_consensus.cc:2379] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 3: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:38:57.020220 19874 raft_consensus.cc:2726] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:38:57.021703 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:57.021754 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:57.006654 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:57.025134 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:57.010439 19704 consensus_queue.cc:1035] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 03:38:57.027516 19917 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:57.028424 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:57.028164 19917 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:57.029346 18069 raft_consensus.cc:2379] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 3: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:38:57.029892 18141 leader_election.cc:302] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:57.030602 19917 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:57.030679 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:57.032013 19948 leader_election.cc:288] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:57.033231 19871 leader_election.cc:288] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:57.034371 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:57.034431 18249 leader_election.cc:302] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:57.035259 18069 raft_consensus.cc:2454] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:38:57.029248 18298 raft_consensus.cc:2454] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3.
I20260807 03:38:57.037767 19496 raft_consensus.cc:2726] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:57.035802 18249 leader_election.cc:302] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:57.038211 18293 consensus_queue.cc:237] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:38:57.038882 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:57.040787 18141 leader_election.cc:302] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:57.041723 19496 raft_consensus.cc:2781] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:57.042114 19496 raft_consensus.cc:482] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:57.042240 18249 leader_election.cc:302] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:57.042462 19496 raft_consensus.cc:3037] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:57.042913 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:57.043692 18069 raft_consensus.cc:3037] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:57.045511 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:57.045961 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:57.046353 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:57.046531 19825 raft_consensus.cc:2726] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:38:57.046653 18193 raft_consensus.cc:3037] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:56.976310 17912 rpcz_store.cc:276] 0807 03:38:54.066747 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:54.068521 (+ 1774us) service_pool.cc:227] Handling call
0807 03:38:56.971838 (+2903317us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 03:38:56.129643 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 03:38:56.130178 (+ 535us) write_op.cc:432] Acquiring schema lock in shared mode
0807 03:38:56.130216 (+ 38us) write_op.cc:435] Acquired schema lock
0807 03:38:56.130233 (+ 17us) tablet.cc:660] Decoding operations
0807 03:38:56.181882 (+ 51649us) write_op.cc:620] Acquiring the partition lock for write op
0807 03:38:56.181945 (+ 63us) write_op.cc:641] Partition lock acquired for write op
0807 03:38:56.181971 (+ 26us) tablet.cc:683] Acquiring locks for 4 operations
0807 03:38:56.182354 (+ 383us) tablet.cc:699] Row locks acquired
0807 03:38:56.182375 (+ 21us) write_op.cc:260] PREPARE: finished
0807 03:38:56.182560 (+ 185us) write_op.cc:270] Start()
0807 03:38:56.182694 (+ 134us) write_op.cc:276] Timestamp: P: 1786073936182505 usec, L: 0
0807 03:38:56.182716 (+ 22us) op_driver.cc:331] REPLICATION: starting
0807 03:38:56.183350 (+ 634us) log.cc:842] Serialized 1717 byte log entry
0807 03:38:56.584659 (+401309us) op_driver.cc:444] REPLICATION: finished
0807 03:38:56.611606 (+ 26947us) write_op.cc:301] APPLY: starting
0807 03:38:56.611700 (+ 94us) tablet.cc:1365] starting BulkCheckPresence
0807 03:38:56.611862 (+ 162us) tablet.cc:1368] finished BulkCheckPresence
0807 03:38:56.612008 (+ 146us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 03:38:56.613588 (+ 1580us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 03:38:56.613801 (+ 213us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0807 03:38:56.613868 (+ 67us) write_op.cc:312] APPLY: finished
0807 03:38:56.614144 (+ 276us) log.cc:842] Serialized 40 byte log entry
0807 03:38:56.614294 (+ 150us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:38:56.614405 (+ 111us) write_op.cc:454] Released schema lock
0807 03:38:56.614455 (+ 50us) write_op.cc:341] FINISH: Updating metrics
0807 03:38:56.614715 (+ 260us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:38:56.614740 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":134625,"reported_tablets":18,"child_traces":[["op",{"apply.queue_time_us":26661,"apply.run_cpu_time_us":3403,"apply.run_wall_time_us":3403,"num_ops":4,"prepare.queue_time_us":3143,"prepare.run_cpu_time_us":3265,"prepare.run_wall_time_us":57217,"replication_time_us":401823,"spinlock_wait_cycles":1766784,"thread_start_us":20801,"threads_started":1}]]}
I20260807 03:38:57.057698 19927 raft_consensus.cc:482] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:57.057967 18249 leader_election.cc:302] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:57.058295 19927 raft_consensus.cc:504] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:57.067739 18070 raft_consensus.cc:1261] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 03:38:57.049659 18140 leader_election.cc:302] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:57.066200 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:57.074873 18081 raft_consensus.cc:2379] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 3: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:38:57.030958 19829 raft_consensus.cc:2781] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:57.064896 19743 raft_consensus.cc:2781] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:57.081596 18249 leader_election.cc:302] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:57.077666 19829 raft_consensus.cc:482] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:57.078012 19917 raft_consensus.cc:2781] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:38:57.080331 19743 raft_consensus.cc:482] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:57.077108 18072 raft_consensus.cc:2454] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:38:57.092594 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:57.102061 19980 raft_consensus.cc:3032] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 03:38:57.102422 19980 raft_consensus.cc:729] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:57.103184 19980 consensus_queue.cc:260] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:57.104460 19980 raft_consensus.cc:3037] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:57.105088 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:57.105813 18075 raft_consensus.cc:2379] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 3: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:38:57.097693 19743 raft_consensus.cc:3037] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:57.107038 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:57.107821 18298 consensus_queue.cc:237] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:38:57.109262 19847 raft_consensus.cc:2781] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:57.109652 19847 raft_consensus.cc:482] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:57.110061 19847 raft_consensus.cc:3037] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:57.110357 18310 raft_consensus.cc:2454] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 5.
I20260807 03:38:57.112367 18305 raft_consensus.cc:2454] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3.
I20260807 03:38:57.117161 18248 leader_election.cc:398] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Vote denied by peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163) with higher term. Message: Invalid argument: T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 4. Current term is 5.
I20260807 03:38:57.117446 18248 leader_election.cc:401] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260807 03:38:57.118121 19719 raft_consensus.cc:3037] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 5
I20260807 03:38:57.119196 19659 raft_consensus.cc:482] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:57.119761 19659 raft_consensus.cc:504] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:57.120759 19583 raft_consensus.cc:504] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:57.121749 19659 leader_election.cc:288] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:57.121927 18191 raft_consensus.cc:1226] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260807 03:38:57.122654 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:57.123292 18079 raft_consensus.cc:3037] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 3
I20260807 03:38:57.128350 18081 raft_consensus.cc:2454] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:38:57.129462 19917 raft_consensus.cc:686] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:57.130473 19917 consensus_queue.cc:237] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:57.132232 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 91554.2us
I20260807 03:38:57.133001 18141 leader_election.cc:302] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:57.132408 17920 catalog_manager.cc:5495] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 1 to 2, leader changed from d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 2 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:57.134672 17920 catalog_manager.cc:5495] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 1 to 2, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 2 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
W20260807 03:38:57.136137 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 70669.7us
I20260807 03:38:57.136245 19797 raft_consensus.cc:2781] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:57.136827 19860 leader_election.cc:288] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:57.138993 19719 raft_consensus.cc:2726] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163) with higher term. Message: Invalid argument: T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 4. Current term is 5.
I20260807 03:38:57.115386 19496 raft_consensus.cc:504] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:57.147460 18193 raft_consensus.cc:2454] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3.
W20260807 03:38:57.148310 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 70759.6us
I20260807 03:38:57.089591 19829 raft_consensus.cc:3037] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:57.150285 18140 leader_election.cc:302] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:57.158273 18079 raft_consensus.cc:2454] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:38:57.160375 19583 leader_election.cc:288] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:57.164839 17920 catalog_manager.cc:5495] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 1 to 3, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 3 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:38:57.166381 19704 raft_consensus.cc:2781] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:57.166833 19704 raft_consensus.cc:482] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:57.167171 19704 raft_consensus.cc:3037] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:57.167452 17920 catalog_manager.cc:5495] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:38:57.169936 19496 leader_election.cc:288] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:57.172384 18141 leader_election.cc:302] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:57.174222 19795 raft_consensus.cc:2781] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:57.174649 19795 raft_consensus.cc:482] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:57.175065 19795 raft_consensus.cc:3037] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:57.158594 19809 raft_consensus.cc:2741] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election decision vote started in defunct term 2: won
I20260807 03:38:57.153363 19797 raft_consensus.cc:482] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:57.180970 19797 raft_consensus.cc:3037] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:57.183035 19704 raft_consensus.cc:504] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:57.183925 19743 raft_consensus.cc:504] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:57.185178 19704 leader_election.cc:288] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:57.186522 19743 leader_election.cc:288] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:57.186569 19973 consensus_queue.cc:1035] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 03:38:57.175673 18141 leader_election.cc:302] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:57.197322 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:57.198068 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:57.202104 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:57.283267 18312 raft_consensus.cc:3037] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:57.200872 18082 raft_consensus.cc:3037] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:38:57.201115 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:57.202768 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:57.195399 18027 leader_election.cc:302] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:38:57.205031 18248 leader_election.cc:302] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:57.188634 18249 leader_election.cc:302] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:57.194172 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:57.202734 19847 raft_consensus.cc:504] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:57.199384 18140 leader_election.cc:302] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:57.282238 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:57.197659 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:57.195529 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:57.297632 18305 raft_consensus.cc:2379] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 3: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:38:57.298693 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:38:57.300488 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 101173us
I20260807 03:38:57.300652 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:57.302868 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:57.194150 19809 raft_consensus.cc:2726] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 03:38:57.192421 19852 leader_election.cc:288] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:57.207739 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:57.194562 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:57.313261 18079 raft_consensus.cc:2354] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 2. Current term is 3.
I20260807 03:38:57.193284 19927 leader_election.cc:288] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:57.314994 18305 raft_consensus.cc:2454] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:38:57.196926 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:57.197575 18069 raft_consensus.cc:2454] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:38:57.326378 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:57.334012 18295 raft_consensus.cc:2379] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 3: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:38:57.335902 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:57.188055 19797 raft_consensus.cc:504] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:57.336572 18080 raft_consensus.cc:3037] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:57.207361 19829 raft_consensus.cc:504] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:57.338809 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 133869us
I20260807 03:38:57.338613 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:57.319351 19973 raft_consensus.cc:2781] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:38:57.324898 18072 raft_consensus.cc:3037] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 3
W20260807 03:38:57.346164 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 150906us
I20260807 03:38:57.347240 18248 leader_election.cc:302] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:57.335903 19832 raft_consensus.cc:2781] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:57.352741 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:57.382946 18309 raft_consensus.cc:3037] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 3
I20260807 03:38:57.383083 18080 raft_consensus.cc:2454] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:38:57.344300 19973 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 03:38:57.384656 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 48001.7us
I20260807 03:38:57.384771 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:57.355219 19860 raft_consensus.cc:2726] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:38:57.356647 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:57.357016 19797 leader_election.cc:288] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:38:57.357447 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 168880us
I20260807 03:38:57.360210 19743 raft_consensus.cc:2781] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:38:57.360363 18027 leader_election.cc:302] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:57.368666 19659 raft_consensus.cc:2781] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:38:57.370782 18248 leader_election.cc:398] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 election: Vote denied by peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163) with higher term. Message: Invalid argument: T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 2. Current term is 3.
I20260807 03:38:57.350697 18310 raft_consensus.cc:2379] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 3: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:38:57.373987 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:57.374157 19583 raft_consensus.cc:2932] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:38:57.377363 18082 raft_consensus.cc:1261] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 03:38:57.378216 18300 raft_consensus.cc:2454] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:38:57.402096 18249 leader_election.cc:302] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:57.386247 18180 raft_consensus.cc:2354] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 2. Current term is 3.
I20260807 03:38:57.384516 18312 raft_consensus.cc:2454] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:38:57.393291 18248 leader_election.cc:401] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260807 03:38:57.394800 19659 raft_consensus.cc:686] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:57.385985 19973 raft_consensus.cc:3037] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:38:57.348696 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
W20260807 03:38:57.402673 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 51535.2us
I20260807 03:38:57.405711 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:57.406220 19832 raft_consensus.cc:482] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:57.406538 19659 consensus_queue.cc:237] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:57.407379 19291 raft_consensus.cc:1261] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 03:38:57.408126 19292 raft_consensus.cc:1261] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 3 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260807 03:38:57.409510 19297 raft_consensus.cc:1261] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 03:38:57.411976 18141 leader_election.cc:302] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:57.414116 18309 raft_consensus.cc:2454] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:38:57.400460 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:57.424974 18249 leader_election.cc:302] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
W20260807 03:38:57.427533 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 56891.3us
I20260807 03:38:57.428994 19704 raft_consensus.cc:3037] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:57.404011 19743 raft_consensus.cc:686] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
W20260807 03:38:57.432691 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 92249.8us
I20260807 03:38:57.416003 19832 raft_consensus.cc:3037] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:57.416023 19871 raft_consensus.cc:2781] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:38:57.434803 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:57.436931 19743 consensus_queue.cc:237] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:57.440016 18140 leader_election.cc:302] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:57.440711 19829 leader_election.cc:288] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:57.441507 19955 raft_consensus.cc:2726] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163) with higher term. Message: Invalid argument: T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 2. Current term is 3.
I20260807 03:38:57.442058 20004 raft_consensus.cc:2781] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:38:57.442216 19871 raft_consensus.cc:686] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:38:57.442586 20004 raft_consensus.cc:686] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:57.443384 19871 consensus_queue.cc:237] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:57.443877 20004 consensus_queue.cc:237] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:57.445983 18140 leader_election.cc:302] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:57.446762 18027 leader_election.cc:302] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:57.401116 19795 raft_consensus.cc:504] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:57.424124 19809 raft_consensus.cc:482] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:57.442211 19496 consensus_queue.cc:1035] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.015s
I20260807 03:38:57.432977 19719 raft_consensus.cc:2726] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:57.473330 19825 consensus_queue.cc:1035] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:38:57.475282 19927 consensus_queue.cc:1035] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:38:57.475744 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:57.476562 18082 raft_consensus.cc:2354] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 2. Current term is 3.
I20260807 03:38:57.458920 18080 raft_consensus.cc:1261] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:38:57.466961 18027 leader_election.cc:302] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:57.468037 18070 raft_consensus.cc:1261] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260807 03:38:57.468318 19617 raft_consensus.cc:2726] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:57.469169 18069 raft_consensus.cc:1261] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 03:38:57.453536 19861 raft_consensus.cc:2781] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:38:57.453697 19874 raft_consensus.cc:2726] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:57.434042 19847 leader_election.cc:288] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:57.481079 18249 leader_election.cc:302] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:57.493553 19829 raft_consensus.cc:482] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:57.494226 18248 leader_election.cc:398] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 election: Vote denied by peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163) with higher term. Message: Invalid argument: T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 2. Current term is 3.
I20260807 03:38:57.498620 18248 leader_election.cc:401] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260807 03:38:57.494624 18081 raft_consensus.cc:1261] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 03:38:57.502334 19977 consensus_queue.cc:1035] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 03:38:57.504442 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:57.497144 19861 raft_consensus.cc:686] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 3 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:57.516956 19617 consensus_queue.cc:1035] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20260807 03:38:57.517005 19480 raft_consensus.cc:482] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:57.521567 19480 raft_consensus.cc:504] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:57.517813 18249 leader_election.cc:302] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:57.488718 19924 consensus_queue.cc:1035] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.026s
I20260807 03:38:57.511799 19847 raft_consensus.cc:3037] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:57.489938 19809 raft_consensus.cc:504] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:57.515385 19955 raft_consensus.cc:2726] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:38:57.539497 19944 raft_consensus.cc:2726] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 03:38:57.542258 19997 log.cc:924] Time spent T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.137s user 0.000s sys 0.001s
I20260807 03:38:57.542753 19291 raft_consensus.cc:2932] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
W20260807 03:38:57.550819 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 139010us
I20260807 03:38:57.551797 18141 leader_election.cc:302] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:57.547185 18249 leader_election.cc:302] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:57.514374 19829 raft_consensus.cc:504] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:57.566082 18311 raft_consensus.cc:2454] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 5.
I20260807 03:38:57.578135 18249 leader_election.cc:302] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:57.535897 19860 consensus_queue.cc:1035] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.032s
I20260807 03:38:57.586292 19617 raft_consensus.cc:2726] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:57.567149 19944 raft_consensus.cc:2726] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:57.595492 18141 leader_election.cc:302] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:57.582389 19980 raft_consensus.cc:2726] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:57.599673 19480 leader_election.cc:288] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:57.600442 19480 raft_consensus.cc:2726] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 03:38:57.606237 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 85827us
I20260807 03:38:57.612799 19948 consensus_queue.cc:1035] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:38:57.614571 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:57.612411 18027 leader_election.cc:302] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:57.616611 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:57.617203 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:57.622645 18190 raft_consensus.cc:2379] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 3: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:38:57.618527 19809 leader_election.cc:288] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:57.620108 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:57.620798 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:57.605958 18072 raft_consensus.cc:2454] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:38:57.598491 19704 raft_consensus.cc:2726] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 03:38:57.614857 19980 raft_consensus.cc:482] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:57.627126 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:57.604298 19795 leader_election.cc:288] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:57.619943 19829 leader_election.cc:288] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:57.637833 18187 raft_consensus.cc:2379] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 3: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:38:57.646843 19874 raft_consensus.cc:2726] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 03:38:57.655479 18249 leader_election.cc:302] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
W20260807 03:38:57.656514 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 44258.8us
I20260807 03:38:57.657634 18027 leader_election.cc:302] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:57.657835 19641 consensus_queue.cc:1035] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 03:38:57.658669 18027 leader_election.cc:302] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:57.658759 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:57.659673 18069 raft_consensus.cc:2379] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 2: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:38:57.631893 19809 raft_consensus.cc:482] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:57.660254 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:57.660516 20018 raft_consensus.cc:2781] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:57.660940 20018 raft_consensus.cc:482] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 03:38:57.661067 20009 log.cc:924] Time spent T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20260807 03:38:57.661253 20018 raft_consensus.cc:3037] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:57.661800 18249 leader_election.cc:302] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:57.662554 19617 raft_consensus.cc:2726] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 03:38:57.663195 19617 raft_consensus.cc:2726] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 03:38:57.659687 18027 leader_election.cc:302] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
W20260807 03:38:57.667857 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 76581.5us
I20260807 03:38:57.668764 18141 leader_election.cc:302] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:57.677201 18191 raft_consensus.cc:2454] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:38:57.681882 19847 raft_consensus.cc:2726] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163) with higher term. Message: Invalid argument: T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 2. Current term is 3.
I20260807 03:38:57.683328 18248 leader_election.cc:302] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:57.634380 19902 raft_consensus.cc:2741] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260807 03:38:57.592417 19861 consensus_queue.cc:237] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:57.697379 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:57.698537 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:57.689554 18141 leader_election.cc:302] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
W20260807 03:38:57.704418 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 44850.9us
I20260807 03:38:57.705032 19617 raft_consensus.cc:2781] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:38:57.714672 18249 leader_election.cc:302] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:57.645623 19980 raft_consensus.cc:504] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:57.718868 19874 raft_consensus.cc:2781] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:38:57.727810 19980 leader_election.cc:288] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:57.660261 19809 raft_consensus.cc:504] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:57.722738 19829 raft_consensus.cc:482] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:57.729979 19809 leader_election.cc:288] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:57.729897 19829 raft_consensus.cc:504] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:38:57.717722 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:57.724040 20011 raft_consensus.cc:482] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:57.724267 19719 raft_consensus.cc:2726] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 03:38:57.724588 18188 raft_consensus.cc:1261] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:38:57.720175 19641 raft_consensus.cc:2726] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:38:57.726070 18069 raft_consensus.cc:1261] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:38:57.700695 19795 raft_consensus.cc:2726] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 03:38:57.727752 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
W20260807 03:38:57.736464 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 47039.3us
I20260807 03:38:57.742897 18141 leader_election.cc:302] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:57.748076 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:57.745464 18249 leader_election.cc:302] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:57.751632 18027 leader_election.cc:398] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Vote denied by peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021) with higher term. Message: Invalid argument: T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 2. Current term is 3.
I20260807 03:38:57.752030 18027 leader_election.cc:401] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260807 03:38:57.753094 18185 raft_consensus.cc:1261] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260807 03:38:57.756769 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:57.759972 18069 raft_consensus.cc:2454] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:38:57.764875 19583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:44650 (request call id 106) took 1798 ms. Trace:
I20260807 03:38:57.765168 19295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:38:57.765899 19295 raft_consensus.cc:2454] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:38:57.766911 18249 leader_election.cc:302] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
W20260807 03:38:57.757112 20006 log.cc:924] Time spent T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.148s user 0.000s sys 0.001s
I20260807 03:38:57.768836 18079 raft_consensus.cc:2379] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 3: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:38:57.760191 18027 leader_election.cc:302] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:38:57.770237 19496 raft_consensus.cc:2726] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:57.743500 19617 raft_consensus.cc:686] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:57.775892 18141 leader_election.cc:302] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:57.772413 18138 leader_election.cc:302] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:57.772185 20005 raft_consensus.cc:2726] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:38:57.794890 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:57.766978 19743 raft_consensus.cc:2726] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:57.759404 19829 leader_election.cc:288] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:38:57.784461 19847 raft_consensus.cc:2932] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:38:57.782763 19853 consensus_queue.cc:1035] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20260807 03:38:57.808954 19902 raft_consensus.cc:2726] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021) with higher term. Message: Invalid argument: T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 2. Current term is 3.
I20260807 03:38:57.809648 19902 raft_consensus.cc:2781] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election won for term 5
I20260807 03:38:57.810115 19902 raft_consensus.cc:686] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:38:57.811069 19902 consensus_queue.cc:237] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:57.765275 19583 rpcz_store.cc:276] 0807 03:38:55.966519 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:55.966651 (+ 132us) service_pool.cc:227] Handling call
0807 03:38:56.261386 (+294735us) spinlock_profiling.cc:243] Waited 36.3 ms on lock 0x7b5800140cb0. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe99213af63 00007fe99219eaba 00007fe9921baa7a 00007fe9921bbada 00007fe9921bac8b 00007fe9921c6084 00007fe9921c5743 00007fe9921bdc1a 00007fe9971016d9 00007fe9907eb184
0807 03:38:56.366166 (+104780us) log.cc:842] Serialized 508 byte log entry
0807 03:38:56.822551 (+456385us) spinlock_profiling.cc:243] Waited 3.33 ms on lock 0x7b54000de6a0. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe992150a4f 00007fe992139545 00007fe99213e771 00007fe99213e6f9 00007fe99213e689 00007fe99213e651 00007fe99213d81d 00007fe9984fb204 00007fe9984fb039 00007fe98f2976b6
0807 03:38:56.865167 (+ 42616us) spinlock_profiling.cc:243] Waited 2.88 ms on lock 0x7b54000de6a0. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe992158416 00007fe992151daa 00007fe992151014 00007fe992139545 00007fe99213e771 00007fe99213e6f9 00007fe99213e689 00007fe99213e651 00007fe99213d81d 00007fe9984fb204
0807 03:38:56.913962 (+ 48795us) spinlock_profiling.cc:243] Waited 25.6 ms on lock 0x7b44000232cc. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f1a9a95 00007fe98f1aa4b2 00007fe991e52511 00007fe991e4de1a 00007fe991e41bee 00007fe991e40f36 00007fe991e40b58 00007fe991e4a551 00007fe991e4a509 00007fe991e4a499
0807 03:38:57.042506 (+128544us) spinlock_profiling.cc:243] Waited 1.08 ms on lock 0x7b54000de6a0. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe99214d771 00007fe992158220 00007fe9921581c8 00007fe992158148 00007fe992158103 00007fe99215738c 00007fe9981c8206 00007fe9981c0808 00007fe99219438b 00007fe992196e80
0807 03:38:57.764780 (+722274us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":7930,"raft.run_cpu_time_us":6183,"raft.run_wall_time_us":780410,"spinlock_wait_cycles":159207040,"thread_start_us":82310,"threads_started":1,"wal-append.queue_time_us":106337}
I20260807 03:38:57.812676 19583 raft_consensus.cc:2781] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:38:57.813174 19583 raft_consensus.cc:482] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:57.813570 19583 raft_consensus.cc:3037] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:38:57.814307 19980 raft_consensus.cc:2781] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:38:57.814913 19980 raft_consensus.cc:686] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:57.815243 18069 raft_consensus.cc:1261] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:38:57.815095 18141 leader_election.cc:302] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
W20260807 03:38:57.828958 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 51510.1us
I20260807 03:38:57.815802 19980 consensus_queue.cc:237] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:57.842869 19874 raft_consensus.cc:686] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:38:57.740975 20011 raft_consensus.cc:504] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:38:57.853335 18141 leader_election.cc:302] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:57.855346 18141 leader_election.cc:302] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:57.861719 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:57.823593 19927 consensus_queue.cc:1035] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20260807 03:38:57.866211 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:57.827335 19825 consensus_queue.cc:1035] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.037s
I20260807 03:38:57.833767 20005 consensus_queue.cc:1035] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
W20260807 03:38:57.869917 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 109841us
I20260807 03:38:57.870616 18027 leader_election.cc:302] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:57.861009 20021 raft_consensus.cc:2726] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 03:38:57.807992 19617 consensus_queue.cc:237] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:57.875785 18249 leader_election.cc:302] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:57.871460 19917 raft_consensus.cc:2781] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 03:38:57.880380 17899 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet af8877c3fbba4ab18765832f1390921f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 03:38:57.861104 19832 raft_consensus.cc:504] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:57.885299 20023 log.cc:924] Time spent T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.111s user 0.000s sys 0.001s
I20260807 03:38:57.887389 18183 raft_consensus.cc:1261] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:38:57.889807 20018 raft_consensus.cc:504] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:57.890931 19973 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:57.891233 20042 consensus_queue.cc:1046] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 03:38:57.876749 20026 raft_consensus.cc:2726] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260807 03:38:57.893739 17899 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 66019.5us
I20260807 03:38:57.894081 20011 leader_election.cc:288] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:57.895352 19917 raft_consensus.cc:686] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:57.897789 20034 raft_consensus.cc:2781] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:38:57.898425 18082 raft_consensus.cc:2932] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
W20260807 03:38:57.927605 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 50904.3us
I20260807 03:38:57.930862 17920 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39494 (request call id 134) took 1012 ms. Trace:
I20260807 03:38:57.946005 18188 raft_consensus.cc:1226] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 4. Current term is 5. Ops: []
I20260807 03:38:57.937160 19948 raft_consensus.cc:2726] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 03:38:57.961424 18180 raft_consensus.cc:1261] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:38:57.944002 19980 raft_consensus.cc:482] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:57.883245 19874 consensus_queue.cc:237] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:57.947140 19797 raft_consensus.cc:2726] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 03:38:57.968830 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:57.899979 19583 raft_consensus.cc:504] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:38:57.970611 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:57.943054 17920 rpcz_store.cc:276] 0807 03:38:56.918442 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:56.918664 (+ 222us) service_pool.cc:227] Handling call
0807 03:38:57.930766 (+1012102us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 03:38:57.429437 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 03:38:57.430233 (+ 796us) write_op.cc:432] Acquiring schema lock in shared mode
0807 03:38:57.430305 (+ 72us) write_op.cc:435] Acquired schema lock
0807 03:38:57.430327 (+ 22us) tablet.cc:660] Decoding operations
0807 03:38:57.496956 (+ 66629us) write_op.cc:620] Acquiring the partition lock for write op
0807 03:38:57.501809 (+ 4853us) write_op.cc:641] Partition lock acquired for write op
0807 03:38:57.501895 (+ 86us) tablet.cc:683] Acquiring locks for 4 operations
0807 03:38:57.502440 (+ 545us) tablet.cc:699] Row locks acquired
0807 03:38:57.502481 (+ 41us) write_op.cc:260] PREPARE: finished
0807 03:38:57.502651 (+ 170us) write_op.cc:270] Start()
0807 03:38:57.502799 (+ 148us) write_op.cc:276] Timestamp: P: 1786073937502590 usec, L: 0
0807 03:38:57.502842 (+ 43us) op_driver.cc:331] REPLICATION: starting
0807 03:38:57.503521 (+ 679us) log.cc:842] Serialized 1717 byte log entry
0807 03:38:57.666562 (+163041us) op_driver.cc:444] REPLICATION: finished
0807 03:38:57.756580 (+ 90018us) write_op.cc:301] APPLY: starting
0807 03:38:57.756670 (+ 90us) tablet.cc:1365] starting BulkCheckPresence
0807 03:38:57.756864 (+ 194us) tablet.cc:1368] finished BulkCheckPresence
0807 03:38:57.756996 (+ 132us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 03:38:57.760496 (+ 3500us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 03:38:57.760819 (+ 323us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0807 03:38:57.760885 (+ 66us) write_op.cc:312] APPLY: finished
0807 03:38:57.761224 (+ 339us) log.cc:842] Serialized 40 byte log entry
0807 03:38:57.761402 (+ 178us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:38:57.761505 (+ 103us) write_op.cc:454] Released schema lock
0807 03:38:57.761563 (+ 58us) write_op.cc:341] FINISH: Updating metrics
0807 03:38:57.761848 (+ 285us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:38:57.761887 (+ 39us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":331,"reported_tablets":10,"child_traces":[["op",{"apply.queue_time_us":89680,"apply.run_cpu_time_us":3184,"apply.run_wall_time_us":5622,"num_ops":4,"prepare.queue_time_us":255494,"prepare.run_cpu_time_us":4453,"prepare.run_wall_time_us":74411,"replication_time_us":163604,"thread_start_us":218804,"threads_started":2}]]}
I20260807 03:38:57.954828 20034 raft_consensus.cc:686] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:38:57.956317 19945 raft_consensus.cc:482] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
W20260807 03:38:57.978303 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 123025us
I20260807 03:38:57.970517 20040 raft_consensus.cc:482] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:57.980700 20018 leader_election.cc:288] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:57.987017 19917 consensus_queue.cc:237] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:58.073531 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 80392.6us
W20260807 03:38:58.077118 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 77525.6us
I20260807 03:38:58.078161 18141 leader_election.cc:302] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:57.992162 19980 raft_consensus.cc:504] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:58.073583 18189 raft_consensus.cc:1226] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 3. Current term is 4. Ops: [3.4-3.4]
W20260807 03:38:58.080761 20006 log.cc:924] Time spent T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
I20260807 03:38:58.013795 19945 raft_consensus.cc:504] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:57.994683 20040 raft_consensus.cc:504] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:57.937372 19955 raft_consensus.cc:2932] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:38:57.992697 20051 consensus_queue.cc:1035] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.147s
I20260807 03:38:58.147677 20052 raft_consensus.cc:2726] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 03:38:58.142870 20042 raft_consensus.cc:482] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:58.133062 19829 raft_consensus.cc:482] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:58.138417 19832 leader_election.cc:288] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:58.186033 18027 leader_election.cc:302] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:58.144193 19973 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:38:58.191197 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 193080us
I20260807 03:38:58.245689 18293 raft_consensus.cc:1261] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:38:58.189713 19832 raft_consensus.cc:482] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:58.251513 20008 raft_consensus.cc:482] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:58.252069 20008 raft_consensus.cc:504] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:58.254185 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:58.254485 20069 raft_consensus.cc:2726] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:58.255065 18310 raft_consensus.cc:2379] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 5: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:38:58.255961 18300 raft_consensus.cc:1226] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260807 03:38:58.256888 19292 raft_consensus.cc:2932] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:38:58.259055 19924 raft_consensus.cc:482] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:58.259675 19924 raft_consensus.cc:504] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:58.267720 18189 raft_consensus.cc:1261] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:38:58.225332 19829 raft_consensus.cc:504] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:58.271545 19829 leader_election.cc:288] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:58.273043 20008 leader_election.cc:288] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:58.177554 20042 raft_consensus.cc:504] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:58.262020 19924 leader_election.cc:288] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:58.243800 19847 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:57260 (request call id 203) took 1248 ms. Trace:
I20260807 03:38:58.314508 19847 rpcz_store.cc:276] 0807 03:38:56.974039 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:57.036533 (+ 62494us) service_pool.cc:227] Handling call
0807 03:38:57.110955 (+ 74422us) log.cc:842] Serialized 508 byte log entry
0807 03:38:58.221899 (+1110944us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":79238,"raft.queue_time_us":70678,"raft.run_cpu_time_us":1320,"raft.run_wall_time_us":62556,"thread_start_us":170013,"threads_started":1,"wal-append.queue_time_us":256172}
I20260807 03:38:58.294044 19871 raft_consensus.cc:3032] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 LEADER]: Stepping down as leader of term 2
I20260807 03:38:58.316767 19871 raft_consensus.cc:729] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:38:58.317891 19871 consensus_queue.cc:260] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:58.319195 19871 raft_consensus.cc:3037] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:58.297158 19980 leader_election.cc:288] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:58.301708 20042 leader_election.cc:288] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:58.189786 20034 consensus_queue.cc:237] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:58.303472 20040 leader_election.cc:288] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:58.305035 18027 leader_election.cc:398] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Vote denied by peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057) with higher term. Message: Invalid argument: T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 2. Current term is 3.
I20260807 03:38:58.305599 20018 consensus_queue.cc:1035] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 03:38:58.265970 19832 raft_consensus.cc:504] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:58.312760 18183 raft_consensus.cc:1261] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:38:58.312855 19945 leader_election.cc:288] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:58.327273 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:58.344775 18082 raft_consensus.cc:3037] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:38:58.345757 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:58.334836 19825 consensus_queue.cc:1046] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:38:58.335390 18075 raft_consensus.cc:1261] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:38:58.336071 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:58.337122 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:58.343559 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:58.358733 18079 raft_consensus.cc:2379] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 4: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:38:58.358333 18082 raft_consensus.cc:2454] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:38:58.369944 18073 raft_consensus.cc:2454] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:38:58.370209 17912 catalog_manager.cc:5495] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 1 to 2, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 2 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
W20260807 03:38:58.374439 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 294234us
I20260807 03:38:58.376027 18082 raft_consensus.cc:1226] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:58.390852 17912 catalog_manager.cc:5495] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 1 to 2, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 2 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:58.391974 17899 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 1b214306f53244f8b6ff2a1a9019abd1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 03:38:58.395488 18027 leader_election.cc:398] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Vote denied by peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057) with higher term. Message: Invalid argument: T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 2. Current term is 3.
I20260807 03:38:58.399453 17912 catalog_manager.cc:5495] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:58.402545 18082 raft_consensus.cc:1261] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260807 03:38:58.405997 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 81380us
I20260807 03:38:58.400521 17912 catalog_manager.cc:5495] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 1 to 2, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 2 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:38:58.408298 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:58.356237 20040 consensus_queue.cc:1035] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.045s
I20260807 03:38:58.411129 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:58.408922 18188 raft_consensus.cc:2454] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:38:58.373176 19743 consensus_queue.cc:1046] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.043s
I20260807 03:38:58.418803 17912 catalog_manager.cc:5495] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 1 to 2, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 2 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:38:58.394980 19795 raft_consensus.cc:482] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:58.423697 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:58.424301 18305 raft_consensus.cc:3037] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:58.425840 18027 leader_election.cc:398] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Vote denied by peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057) with higher term. Message: Invalid argument: T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 2. Current term is 3.
I20260807 03:38:58.424197 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:58.426128 18027 leader_election.cc:401] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260807 03:38:58.428296 18140 leader_election.cc:302] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:58.431566 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:58.323937 19297 raft_consensus.cc:2932] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
W20260807 03:38:58.473935 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 48234.4us
I20260807 03:38:58.406149 19583 leader_election.cc:288] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:38:58.478188 18298 raft_consensus.cc:1261] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:38:58.479974 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:58.486055 18027 leader_election.cc:302] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:58.486896 19871 raft_consensus.cc:2726] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 03:38:58.487350 18027 leader_election.cc:302] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:58.488040 19945 raft_consensus.cc:482] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:58.489347 19945 raft_consensus.cc:504] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:58.490718 19659 raft_consensus.cc:482] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:58.489702 18249 leader_election.cc:302] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:58.491387 19945 leader_election.cc:288] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:58.486302 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:58.434423 17912 catalog_manager.cc:5495] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 1 to 2, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 2 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:38:58.492959 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:58.494414 19871 raft_consensus.cc:2726] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 03:38:58.491957 19945 raft_consensus.cc:2781] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:38:58.495579 19945 raft_consensus.cc:482] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:58.496080 19945 raft_consensus.cc:3037] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:38:58.499800 18305 raft_consensus.cc:2454] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:38:58.500168 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:58.500721 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:58.500752 18188 raft_consensus.cc:2454] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:38:58.501729 18305 raft_consensus.cc:2454] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:38:58.496701 19291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331"
I20260807 03:38:58.504599 18073 raft_consensus.cc:1226] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 3. Current term is 4. Ops: [3.4-3.4]
I20260807 03:38:58.478667 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:58.513909 18295 raft_consensus.cc:2454] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:38:58.497197 18027 leader_election.cc:302] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:58.517318 18249 leader_election.cc:302] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
W20260807 03:38:58.517931 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 53586.9us
I20260807 03:38:58.515347 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:58.501588 19980 raft_consensus.cc:2781] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:58.479357 20018 raft_consensus.cc:2726] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057) with higher term. Message: Invalid argument: T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 2. Current term is 3.
I20260807 03:38:58.548074 20077 raft_consensus.cc:2781] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:38:58.432385 19829 consensus_queue.cc:1035] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.105s
I20260807 03:38:58.474913 19480 consensus_queue.cc:1046] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.077s
I20260807 03:38:58.518783 19743 raft_consensus.cc:482] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
W20260807 03:38:58.615475 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 118363us
I20260807 03:38:58.537217 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:58.603307 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:58.549002 19874 raft_consensus.cc:2726] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:38:58.591558 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:58.613719 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:58.619264 18295 raft_consensus.cc:2379] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 3: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:38:58.619527 18027 leader_election.cc:398] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Vote denied by peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057) with higher term. Message: Invalid argument: T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 2. Current term is 3.
I20260807 03:38:58.617475 18080 raft_consensus.cc:2454] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:38:58.609754 19832 leader_election.cc:288] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:58.624872 19240 leader_election.cc:302] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:58.625243 20062 consensus_queue.cc:1035] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 03:38:58.610420 18310 raft_consensus.cc:2379] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 3: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:38:58.625875 20044 raft_consensus.cc:2726] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:58.627568 20068 consensus_queue.cc:1035] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 03:38:58.518550 19659 raft_consensus.cc:504] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:58.631336 19659 leader_election.cc:288] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:58.633337 17920 catalog_manager.cc:5495] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 reported cstate change: config changed from index 3 to 4, VOTER 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) evicted. New cstate: current_term: 1 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:38:58.634964 17920 catalog_manager.cc:5495] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 reported cstate change: term changed from 2 to 3, leader changed from <none> to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 3 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:38:58.622915 19743 raft_consensus.cc:504] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:58.636854 18072 raft_consensus.cc:1261] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260807 03:38:58.625911 19240 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 70141.5us
I20260807 03:38:58.627859 18310 raft_consensus.cc:1261] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:38:58.638365 19743 leader_election.cc:288] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:58.641986 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:58.625377 18295 raft_consensus.cc:1261] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:38:58.623474 20018 consensus_queue.cc:1046] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20260807 03:38:58.643932 20018 raft_consensus.cc:3032] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 LEADER]: Stepping down as leader of term 2
I20260807 03:38:58.644289 20018 raft_consensus.cc:729] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:38:58.645071 20018 consensus_queue.cc:260] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:58.646175 20018 raft_consensus.cc:3037] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:58.474308 19795 raft_consensus.cc:504] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:38:58.594838 19917 raft_consensus.cc:482] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:58.648512 19917 raft_consensus.cc:504] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:38:58.649061 18069 raft_consensus.cc:1261] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:38:58.649864 19795 leader_election.cc:288] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:38:58.589095 20077 raft_consensus.cc:482] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:58.651029 20077 raft_consensus.cc:3037] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:38:58.651624 19917 leader_election.cc:288] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
W20260807 03:38:58.654028 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 132039us
I20260807 03:38:58.580454 19980 raft_consensus.cc:482] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:58.670621 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:58.671195 18191 raft_consensus.cc:2454] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:38:58.657912 19291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:38:58.678702 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:58.679527 18079 raft_consensus.cc:2454] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
I20260807 03:38:58.682252 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:58.682950 18300 raft_consensus.cc:2454] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:38:58.665465 19980 raft_consensus.cc:3037] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:58.683532 18249 leader_election.cc:302] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:58.686010 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:58.709118 19292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:38:58.687515 19917 consensus_queue.cc:1035] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.025s
I20260807 03:38:58.664844 19874 consensus_queue.cc:1035] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.039s
I20260807 03:38:58.661062 19945 raft_consensus.cc:504] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:58.719439 18309 raft_consensus.cc:2379] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 4: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:38:58.723129 18071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33254 (request call id 859) took 1098 ms. Trace:
I20260807 03:38:58.723585 18071 rpcz_store.cc:276] 0807 03:38:57.625379 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:57.628145 (+ 2766us) service_pool.cc:227] Handling call
0807 03:38:57.772447 (+144302us) spinlock_profiling.cc:243] Waited 23.4 ms on lock 0x7b5800001298. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f25f851 00007fe98f25f702 00007fe99814fb09 00007fe99708b730 00007fe99715179b 00007fe997151821 00007fe9971152b0 00007fe9971007f9 00007fe9971002a2 00007fe9907e6684
0807 03:38:57.797264 (+ 24817us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:38:57.854491 (+ 57227us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:38:57.855898 (+ 1407us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:38:58.303016 (+447118us) spinlock_profiling.cc:243] Waited 1.66 ms on lock 0x7b2c00112ec8. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe990ba2e11 00007fe990bb2442 00007fe9921fac11 00007fe9921bf58a 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772
0807 03:38:58.303792 (+ 776us) log.cc:842] Serialized 508 byte log entry
0807 03:38:58.405548 (+101756us) raft_consensus.cc:1631] Marking committed up to 3
0807 03:38:58.412177 (+ 6629us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:38:58.417327 (+ 5150us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:38:58.721167 (+303840us) raft_consensus.cc:1671] finished
0807 03:38:58.721198 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:38:58.721342 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":57738368,"thread_start_us":91530,"threads_started":1,"wal-append.queue_time_us":282907}
I20260807 03:38:58.727118 19847 raft_consensus.cc:482] T af8877c3fbba4ab18765832f1390921f P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:58.727862 19847 raft_consensus.cc:504] T af8877c3fbba4ab18765832f1390921f P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:38:58.730890 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:58.731949 18184 raft_consensus.cc:2379] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 3: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:38:58.680699 19743 raft_consensus.cc:3032] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 03:38:58.733297 19743 raft_consensus.cc:729] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:58.734393 19743 consensus_queue.cc:260] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:58.735709 19743 raft_consensus.cc:3037] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:38:58.707350 20075 raft_consensus.cc:2781] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:38:58.737982 20075 raft_consensus.cc:482] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:58.738360 20075 raft_consensus.cc:3037] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
W20260807 03:38:58.741735 19923 log.cc:924] Time spent T 00000000000000000000000000000000 P 8da2065a77024f35bdfc0dc3adb83f69: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20260807 03:38:58.744339 19583 raft_consensus.cc:3032] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:58.744758 19583 raft_consensus.cc:729] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:58.762482 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:58.747515 19583 consensus_queue.cc:260] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:58.721269 19945 leader_election.cc:288] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:58.801191 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:58.806715 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:58.807439 18191 raft_consensus.cc:2454] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3.
I20260807 03:38:58.761003 19825 raft_consensus.cc:482] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:58.813114 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:58.811973 19825 raft_consensus.cc:504] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:58.815923 19825 leader_election.cc:288] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:58.815702 20077 raft_consensus.cc:504] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:58.817787 18298 raft_consensus.cc:2354] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 4. Current term is 5.
I20260807 03:38:58.817903 20077 leader_election.cc:288] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:58.810559 20073 raft_consensus.cc:482] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:58.818960 20073 raft_consensus.cc:504] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:58.785280 19955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:57260 (request call id 202) took 1801 ms. Trace:
I20260807 03:38:58.820444 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:58.820919 20073 leader_election.cc:288] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:58.821730 18300 raft_consensus.cc:1261] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260807 03:38:58.823835 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 50067us
I20260807 03:38:58.824848 18311 raft_consensus.cc:1261] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:38:58.828581 18191 raft_consensus.cc:1226] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260807 03:38:58.832362 18027 leader_election.cc:302] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:58.810226 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:58.835530 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:58.842962 18141 leader_election.cc:302] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:58.844591 18141 leader_election.cc:302] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:58.847167 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:58.847736 18293 raft_consensus.cc:3037] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:58.848199 18141 leader_election.cc:302] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:58.827818 19583 raft_consensus.cc:3037] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:58.849079 19480 raft_consensus.cc:2781] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:38:58.849220 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:58.833698 20049 raft_consensus.cc:482] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:58.852825 19917 raft_consensus.cc:2726] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:58.853015 19795 consensus_queue.cc:1035] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:58.853626 19871 consensus_queue.cc:1046] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:58.855355 18141 leader_election.cc:302] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:58.857762 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:58.863343 18191 raft_consensus.cc:2454] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:38:58.858754 19480 raft_consensus.cc:686] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 2 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:38:58.842818 19471 raft_consensus.cc:3032] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:38:58.865633 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:58.868130 18180 raft_consensus.cc:2454] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
I20260807 03:38:58.841285 19874 raft_consensus.cc:2726] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 03:38:58.865860 18141 leader_election.cc:302] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:58.862121 19704 raft_consensus.cc:2781] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:58.870010 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:58.873116 18249 leader_election.cc:302] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:58.841404 19847 leader_election.cc:288] T af8877c3fbba4ab18765832f1390921f P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:38:58.875972 18311 raft_consensus.cc:1261] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:38:58.877477 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:58.879531 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:58.882439 18249 leader_election.cc:302] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:58.867193 20052 raft_consensus.cc:2781] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:58.869581 19471 raft_consensus.cc:729] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:58.857110 20049 raft_consensus.cc:504] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:58.872668 19704 raft_consensus.cc:482] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:58.887336 18310 raft_consensus.cc:1226] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260807 03:38:58.891808 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:58.820313 19955 rpcz_store.cc:276] 0807 03:38:56.973585 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:56.982684 (+ 9099us) service_pool.cc:227] Handling call
0807 03:38:57.077029 (+ 94345us) log.cc:842] Serialized 508 byte log entry
0807 03:38:57.475052 (+398023us) spinlock_profiling.cc:243] Waited 28.3 ms on lock 0x7b180037c2a0. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe9981368b1 00007fe998143661 00007fe98f297764 00007fe98f29ab11 00007fe98f29aac9 00007fe98f29aa59 00007fe98f29aa21 00007fe98f299d1d 00007fe9984fb204 00007fe9984fb039 00007fe98f279dc6
0807 03:38:58.774563 (+1299511us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":47404,"raft.queue_time_us":56471,"raft.run_cpu_time_us":1374,"raft.run_wall_time_us":60120,"spinlock_wait_cycles":65010048,"thread_start_us":164887,"threads_started":1,"wal-append.queue_time_us":274497,"wal-append.run_cpu_time_us":3257,"wal-append.run_wall_time_us":1223203}
I20260807 03:38:58.896652 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:58.903105 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:58.903528 18191 raft_consensus.cc:3037] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 5
I20260807 03:38:58.872566 19829 raft_consensus.cc:2726] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 03:38:58.864396 19980 raft_consensus.cc:504] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:58.903681 18180 raft_consensus.cc:2454] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:38:58.887405 19496 raft_consensus.cc:2781] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:38:58.907545 18249 leader_election.cc:302] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:58.898914 19704 raft_consensus.cc:3037] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:58.888828 20052 raft_consensus.cc:482] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:58.888335 18069 raft_consensus.cc:2454] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:38:58.899107 20078 raft_consensus.cc:2781] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:38:58.918197 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:58.894260 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:58.916980 17899 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 071a939ead594d38851e3b554bfc4d24 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 03:38:58.916580 18295 raft_consensus.cc:1226] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:58.923029 18190 raft_consensus.cc:1226] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260807 03:38:58.912197 19496 raft_consensus.cc:482] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:58.917167 20052 raft_consensus.cc:3037] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:58.920609 20078 raft_consensus.cc:482] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:58.963522 18295 raft_consensus.cc:1261] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260807 03:38:58.966068 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:58.966907 18079 raft_consensus.cc:3037] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:38:58.969460 19948 raft_consensus.cc:482] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:58.970108 19948 raft_consensus.cc:504] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:58.942083 19496 raft_consensus.cc:3037] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:58.944339 20077 raft_consensus.cc:2781] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:38:58.974301 18075 raft_consensus.cc:1261] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:38:58.969607 18300 raft_consensus.cc:1261] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260807 03:38:58.976415 18191 raft_consensus.cc:2454] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:38:58.976758 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:58.983871 20077 raft_consensus.cc:482] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:38:58.983500 18073 raft_consensus.cc:2932] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:38:58.984171 19980 leader_election.cc:288] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:58.986609 20049 leader_election.cc:288] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:58.962911 20078 raft_consensus.cc:3037] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:38:58.992702 18295 raft_consensus.cc:1261] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:38:58.992689 19948 leader_election.cc:288] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:58.995954 18249 leader_election.cc:302] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:58.984329 20077 raft_consensus.cc:3037] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:38:59.002558 18079 raft_consensus.cc:2454] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
W20260807 03:38:59.002876 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 132538us
I20260807 03:38:59.006839 18141 leader_election.cc:302] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:59.008878 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:38:59.009720 18191 raft_consensus.cc:2454] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:38:59.011402 18249 leader_election.cc:302] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:38:59.012694 19853 consensus_queue.cc:1046] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:38:59.012897 20075 raft_consensus.cc:504] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:59.013846 18248 leader_election.cc:302] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:59.015039 20075 leader_election.cc:288] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:59.015354 20091 raft_consensus.cc:2726] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:38:59.001438 20072 raft_consensus.cc:2781] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 03:38:59.017500 18305 raft_consensus.cc:1226] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 3. Current term is 4. Ops: []
I20260807 03:38:59.023237 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:58.937126 19480 consensus_queue.cc:237] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:59.004272 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:59.002446 19291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:38:59.032166 18293 raft_consensus.cc:2454] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3.
I20260807 03:38:59.016000 20091 consensus_queue.cc:1035] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20260807 03:38:59.034960 18248 leader_election.cc:302] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:59.021184 19924 raft_consensus.cc:2781] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:59.020962 20049 raft_consensus.cc:2726] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 03:38:59.053674 18185 raft_consensus.cc:3037] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:38:59.043193 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:59.028712 20072 raft_consensus.cc:686] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:38:59.024178 19853 consensus_queue.cc:1035] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
W20260807 03:38:59.042258 20093 log.cc:924] Time spent T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.609s user 0.009s sys 0.023s
I20260807 03:38:59.067444 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:59.065896 18190 raft_consensus.cc:1261] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:38:59.070082 20072 consensus_queue.cc:237] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:59.064069 20049 consensus_queue.cc:1035] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20260807 03:38:59.044342 19924 raft_consensus.cc:482] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:38:59.076529 19924 raft_consensus.cc:3037] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:59.077193 20111 consensus_queue.cc:1035] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:59.082916 18312 raft_consensus.cc:2396] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20260807 03:38:59.083158 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 177399us
I20260807 03:38:59.084278 18027 leader_election.cc:302] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:59.084981 20062 raft_consensus.cc:2741] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260807 03:38:59.085295 20042 consensus_queue.cc:1046] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:59.043955 19617 raft_consensus.cc:2726] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 03:38:59.091457 19496 raft_consensus.cc:504] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:59.093048 18193 raft_consensus.cc:1226] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 3. Current term is 4. Ops: [3.4-3.4]
W20260807 03:38:59.061694 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 83316.1us
I20260807 03:38:59.095875 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:59.087643 20078 raft_consensus.cc:504] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:59.097218 19924 raft_consensus.cc:504] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:59.098237 20077 raft_consensus.cc:504] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:59.111047 18188 raft_consensus.cc:2454] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 2.
I20260807 03:38:59.133127 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:59.137548 18027 leader_election.cc:302] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:38:59.138960 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:59.142578 18246 leader_election.cc:302] T af8877c3fbba4ab18765832f1390921f P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:38:59.143697 20077 leader_election.cc:288] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:59.143776 18072 raft_consensus.cc:1261] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:38:59.144284 20077 raft_consensus.cc:2726] T af8877c3fbba4ab18765832f1390921f P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:59.144503 20078 leader_election.cc:288] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:59.145586 19948 consensus_queue.cc:1046] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:38:59.146059 19924 leader_election.cc:288] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:59.146443 19832 consensus_queue.cc:1035] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:38:59.147665 19832 raft_consensus.cc:2781] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:38:59.136499 20062 consensus_queue.cc:1035] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.013s
I20260807 03:38:59.135950 20068 consensus_queue.cc:1046] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20260807 03:38:59.130821 19871 consensus_queue.cc:1035] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20260807 03:38:59.148074 19832 raft_consensus.cc:686] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:38:59.161207 19832 consensus_queue.cc:237] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:59.161813 19496 leader_election.cc:288] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:59.148177 18310 raft_consensus.cc:1226] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260807 03:38:59.164270 18079 raft_consensus.cc:1261] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260807 03:38:59.148739 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:59.166756 18190 raft_consensus.cc:3037] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 4
I20260807 03:38:59.149214 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:59.207921 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:59.221261 18140 leader_election.cc:302] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d
W20260807 03:38:59.221524 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 186624us
I20260807 03:38:59.233693 18180 raft_consensus.cc:1226] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 2 LEADER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 1. Current term is 2. Ops: []
W20260807 03:38:59.245375 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 197938us
I20260807 03:38:59.151875 19659 raft_consensus.cc:482] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:59.249027 19659 raft_consensus.cc:504] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:59.151763 20068 raft_consensus.cc:3032] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:38:59.158022 20034 raft_consensus.cc:3032] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 LEADER]: Stepping down as leader of term 2
I20260807 03:38:59.157469 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:59.252876 18248 leader_election.cc:302] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
W20260807 03:38:59.253930 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 116500us
I20260807 03:38:59.256666 18027 leader_election.cc:302] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:59.230990 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:38:59.231869 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:59.247457 19795 raft_consensus.cc:2726] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:59.271519 18305 raft_consensus.cc:1261] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:38:59.172065 19704 raft_consensus.cc:504] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:59.252888 20068 raft_consensus.cc:729] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:38:59.279345 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:38:59.172305 20052 raft_consensus.cc:504] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:59.254588 20062 raft_consensus.cc:482] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:59.286902 18073 raft_consensus.cc:2379] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 3: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:38:59.253034 20034 raft_consensus.cc:729] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:38:59.260705 19874 raft_consensus.cc:2726] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:38:59.294476 18305 raft_consensus.cc:1226] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260807 03:38:59.299515 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:59.301921 18248 leader_election.cc:302] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:59.302081 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:59.302659 18075 raft_consensus.cc:3037] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:38:59.256258 19480 consensus_queue.cc:1035] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20260807 03:38:59.277143 18189 raft_consensus.cc:3037] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:59.281451 20076 raft_consensus.cc:482] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:59.264665 20056 consensus_queue.cc:1035] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20260807 03:38:59.305284 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:59.295971 18080 raft_consensus.cc:2932] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:38:59.314123 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:59.152258 19471 consensus_queue.cc:260] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:38:59.280915 20078 raft_consensus.cc:2781] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:38:59.305774 20091 raft_consensus.cc:2726] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 03:38:59.312839 18191 raft_consensus.cc:1226] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 1. Current term is 2. Ops: []
I20260807 03:38:59.320890 18082 raft_consensus.cc:2379] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 3: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:38:59.278654 20018 consensus_queue.cc:1046] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
I20260807 03:38:59.325765 18190 raft_consensus.cc:2454] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
I20260807 03:38:59.327824 19924 raft_consensus.cc:482] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:59.328472 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:38:59.306562 18183 raft_consensus.cc:3037] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
W20260807 03:38:59.321493 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 65780.1us
I20260807 03:38:59.334957 18185 raft_consensus.cc:1261] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260807 03:38:59.287669 20042 consensus_queue.cc:1046] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20260807 03:38:59.288990 20049 consensus_queue.cc:1035] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
I20260807 03:38:59.343775 18141 leader_election.cc:302] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:59.320963 18310 raft_consensus.cc:1226] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 1. Current term is 2. Ops: []
W20260807 03:38:59.357378 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 103209us
I20260807 03:38:59.357895 18248 leader_election.cc:302] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
W20260807 03:38:59.452636 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 108984us
W20260807 03:38:59.452420 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 151652us
I20260807 03:38:59.455230 18141 leader_election.cc:302] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
W20260807 03:38:59.455399 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 97592.1us
I20260807 03:38:59.455224 18027 leader_election.cc:302] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
W20260807 03:38:59.459374 18246 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.132:41381, user_credentials={real_user=slave}} blocked reactor thread for 150307us
I20260807 03:38:59.323247 20078 raft_consensus.cc:482] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:38:59.453866 18140 leader_election.cc:302] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:59.461551 18141 leader_election.cc:302] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49
I20260807 03:38:59.295809 20062 raft_consensus.cc:504] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:59.328374 19924 raft_consensus.cc:504] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:59.451676 18297 raft_consensus.cc:1226] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20260807 03:38:59.479591 20114 log.cc:924] Time spent T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.206s user 0.000s sys 0.001s
W20260807 03:38:59.481249 20118 log.cc:924] Time spent T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.179s user 0.000s sys 0.001s
I20260807 03:38:59.481027 19980 consensus_queue.cc:1035] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:38:59.338815 20126 consensus_queue.cc:1035] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.153s
I20260807 03:38:59.483250 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
W20260807 03:38:59.490015 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 181271us
I20260807 03:38:59.464929 19945 raft_consensus.cc:2726] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 03:38:59.319856 20076 raft_consensus.cc:504] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:59.464720 19917 raft_consensus.cc:2741] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260807 03:38:59.468508 19583 raft_consensus.cc:2726] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:38:59.465380 20078 raft_consensus.cc:3037] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:59.469623 20104 raft_consensus.cc:2726] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 03:38:59.526856 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 71719.8us
I20260807 03:38:59.485369 19480 raft_consensus.cc:2726] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 03:38:59.506057 20077 raft_consensus.cc:482] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:59.338642 20034 consensus_queue.cc:260] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:38:59.539740 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 65430.2us
I20260807 03:38:59.513352 19704 leader_election.cc:288] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:59.551692 18141 leader_election.cc:302] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:59.336118 20068 consensus_queue.cc:260] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:59.555056 18141 leader_election.cc:398] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021) with higher term. Message: Invalid argument: T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 4. Current term is 5.
I20260807 03:38:59.561621 18141 leader_election.cc:401] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260807 03:38:59.564031 18141 leader_election.cc:302] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:38:59.547004 19480 raft_consensus.cc:482] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 03:38:59.569365 20131 log.cc:924] Time spent T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.100s user 0.001s sys 0.000s
I20260807 03:38:59.569249 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:38:59.544845 20135 raft_consensus.cc:2781] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:38:59.573630 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:59.574498 18310 raft_consensus.cc:2379] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 3: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:38:59.580711 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:38:59.581681 18309 raft_consensus.cc:2354] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3.
I20260807 03:38:59.560842 20052 leader_election.cc:288] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:59.571259 20124 raft_consensus.cc:3037] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:38:59.548794 20077 raft_consensus.cc:504] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:59.570834 20056 raft_consensus.cc:482] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:59.557961 19829 consensus_queue.cc:1035] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
W20260807 03:38:59.629740 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 75678us
W20260807 03:38:59.631932 20127 log.cc:924] Time spent T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.149s user 0.000s sys 0.001s
W20260807 03:38:59.629396 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 99669.6us
W20260807 03:38:59.636580 20115 log.cc:924] Time spent T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.308s user 0.000s sys 0.002s
I20260807 03:38:59.575673 19795 raft_consensus.cc:2726] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021) with higher term. Message: Invalid argument: T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 4. Current term is 5.
I20260807 03:38:59.562861 19704 consensus_queue.cc:1035] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20260807 03:38:59.598423 20068 raft_consensus.cc:3037] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:38:59.658514 18187 raft_consensus.cc:1261] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:38:59.598388 20034 raft_consensus.cc:3037] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:38:59.570899 20042 consensus_queue.cc:1035] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.058s
I20260807 03:38:59.623212 20135 raft_consensus.cc:686] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:38:59.628508 19917 raft_consensus.cc:2726] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 03:38:59.576144 19480 raft_consensus.cc:504] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:59.654740 20133 raft_consensus.cc:482] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:59.637518 18189 raft_consensus.cc:2454] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
I20260807 03:38:59.657217 20052 raft_consensus.cc:482] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:59.779872 18185 raft_consensus.cc:1226] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 1. Current term is 2. Ops: []
W20260807 03:38:59.678395 20125 log.cc:924] Time spent T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.335s user 0.001s sys 0.000s
I20260807 03:38:59.794559 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
W20260807 03:38:59.866353 18075 consensus_meta.cc:287] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 0.561s user 0.009s sys 0.000s
I20260807 03:38:59.866845 18075 raft_consensus.cc:2454] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
I20260807 03:38:59.868949 18248 leader_election.cc:302] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
W20260807 03:38:59.810086 20128 log.cc:924] Time spent T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.439s user 0.000s sys 0.001s
I20260807 03:38:59.909296 20135 consensus_queue.cc:237] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:59.646346 20056 raft_consensus.cc:504] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:38:59.912647 20152 raft_consensus.cc:482] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:38:59.921133 19659 leader_election.cc:288] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:59.938648 20077 leader_election.cc:288] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:38:59.672721 20144 consensus_queue.cc:1046] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.318s
I20260807 03:38:59.948725 19874 consensus_queue.cc:1035] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
W20260807 03:38:59.902464 20113 log.cc:924] Time spent T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.208s user 0.000s sys 0.000s
I20260807 03:38:59.941781 19480 leader_election.cc:288] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:00.057888 20076 leader_election.cc:288] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:00.058012 20154 raft_consensus.cc:482] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:38:59.850214 20133 raft_consensus.cc:504] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:00.071978 20144 raft_consensus.cc:482] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:00.072711 20133 leader_election.cc:288] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:00.058531 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 216950us
I20260807 03:39:00.059022 20111 raft_consensus.cc:482] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:00.072525 20144 raft_consensus.cc:504] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:00.073501 20111 raft_consensus.cc:504] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:00.075870 20111 leader_election.cc:288] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:00.060261 20056 leader_election.cc:288] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:00.076443 20161 raft_consensus.cc:482] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:00.076965 20161 raft_consensus.cc:504] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:00.078860 20161 leader_election.cc:288] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:38:59.810721 20052 raft_consensus.cc:504] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:00.067492 18141 leader_election.cc:398] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Vote denied by peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021) with higher term. Message: Invalid argument: T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3.
I20260807 03:39:00.070431 20154 raft_consensus.cc:504] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:00.060182 19659 raft_consensus.cc:2781] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 03:39:00.062870 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:00.085628 18141 leader_election.cc:401] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260807 03:39:00.098973 20062 leader_election.cc:288] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:00.075428 20144 leader_election.cc:288] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:00.093649 20154 leader_election.cc:288] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:00.124219 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 56899.7us
I20260807 03:39:00.127894 18141 leader_election.cc:302] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
W20260807 03:39:00.197556 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 69749.8us
I20260807 03:39:00.123004 19704 raft_consensus.cc:482] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:00.122395 19659 raft_consensus.cc:686] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
W20260807 03:39:00.165360 18069 consensus_meta.cc:287] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 0.877s user 0.007s sys 0.003s
I20260807 03:39:00.126490 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:00.050592 20152 raft_consensus.cc:504] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:00.217136 20161 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:00.218309 20091 raft_consensus.cc:482] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:00.253113 20156 raft_consensus.cc:482] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:00.253866 20173 raft_consensus.cc:482] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:00.254298 19659 consensus_queue.cc:237] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:00.236719 18069 raft_consensus.cc:2454] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:39:00.255134 20172 raft_consensus.cc:2726] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:00.253893 19291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
W20260807 03:39:00.294494 18027 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.662s user 0.054s sys 0.157s
W20260807 03:39:00.299221 18027 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.667s user 0.054s sys 0.157s
W20260807 03:39:00.299584 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 667715us
W20260807 03:39:00.299582 18082 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.637s user 0.006s sys 0.004s
W20260807 03:39:00.300693 18082 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.638s user 0.006s sys 0.004s
W20260807 03:39:00.295171 18249 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.745s user 0.012s sys 0.003s
W20260807 03:39:00.301400 18249 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.751s user 0.012s sys 0.003s
W20260807 03:39:00.296221 18250 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.738s user 0.005s sys 0.005s
W20260807 03:39:00.303869 18250 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.746s user 0.005s sys 0.005s
I20260807 03:39:00.243078 19704 raft_consensus.cc:504] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:00.325711 18309 raft_consensus.cc:1226] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:00.269551 20161 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:00.321089 18184 consensus_meta.cc:287] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 1.048s user 0.011s sys 0.000s
W20260807 03:39:00.328733 18297 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.745s user 0.000s sys 0.011s
I20260807 03:39:00.320417 20062 raft_consensus.cc:482] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:00.271790 20091 raft_consensus.cc:504] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:00.304505 18069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33268 (request call id 832) took 1123 ms. Trace:
W20260807 03:39:00.473973 18297 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.899s user 0.001s sys 0.020s
I20260807 03:39:00.465492 19291 raft_consensus.cc:2396] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260807 03:39:00.273334 20156 raft_consensus.cc:504] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:00.466645 20052 leader_election.cc:288] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:00.468346 18248 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.599s user 0.006s sys 0.014s
I20260807 03:39:00.405594 18184 raft_consensus.cc:2454] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:39:00.382462 20152 leader_election.cc:288] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:00.510514 20166 log.cc:924] Time spent T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.239s user 0.000s sys 0.000s
I20260807 03:39:00.531358 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:00.277184 20173 raft_consensus.cc:504] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:00.532244 18082 raft_consensus.cc:2379] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5c9b5062bdfa46af9396f1a85025a331 in current term 4: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
W20260807 03:39:00.533012 20165 log.cc:924] Time spent T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.136s user 0.000s sys 0.000s
I20260807 03:39:00.533823 20173 leader_election.cc:288] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:00.511039 20177 raft_consensus.cc:482] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
W20260807 03:39:00.535141 20124 consensus_meta.cc:287] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 0.906s user 0.010s sys 0.000s
I20260807 03:39:00.534817 20177 raft_consensus.cc:504] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:00.535542 20124 raft_consensus.cc:2726] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:00.537101 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:00.537424 20177 leader_election.cc:288] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:00.538086 20177 raft_consensus.cc:482] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:00.512836 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:00.527082 18248 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.657s user 0.007s sys 0.014s
I20260807 03:39:00.465348 18069 rpcz_store.cc:276] 0807 03:38:59.174056 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:59.181074 (+ 7018us) service_pool.cc:227] Handling call
0807 03:39:00.294742 (+1113668us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 03:39:00.479365 20144 raft_consensus.cc:482] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:00.512310 20169 raft_consensus.cc:482] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:00.542490 18075 raft_consensus.cc:2396] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 03:39:00.382772 19874 consensus_queue.cc:1046] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.122s
I20260807 03:39:00.538405 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:00.543866 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 674954us
I20260807 03:39:00.481789 20062 raft_consensus.cc:504] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:00.560302 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:00.561036 18082 raft_consensus.cc:2454] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:00.562292 19704 leader_election.cc:288] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:00.546548 18184 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44660 (request call id 982) took 1391 ms. Trace:
I20260807 03:39:00.562940 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:00.563030 18184 rpcz_store.cc:276] 0807 03:38:59.148376 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:59.157150 (+ 8774us) service_pool.cc:227] Handling call
0807 03:39:00.540148 (+1382998us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 03:39:00.564441 18183 consensus_meta.cc:287] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 1.112s user 0.011s sys 0.000s
I20260807 03:39:00.564713 20161 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:00.537132 20091 leader_election.cc:288] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:00.562942 20156 leader_election.cc:288] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
W20260807 03:39:00.569576 18073 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.084s user 0.005s sys 0.022s
W20260807 03:39:00.577272 18073 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.092s user 0.005s sys 0.022s
I20260807 03:39:00.577736 18079 raft_consensus.cc:1261] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 03:39:00.580528 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:00.558159 20144 raft_consensus.cc:504] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:00.582721 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:00.567188 18183 raft_consensus.cc:2454] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3.
I20260807 03:39:00.587569 19829 raft_consensus.cc:2726] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021) with higher term. Message: Invalid argument: T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3.
I20260807 03:39:00.590333 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:00.590391 20152 raft_consensus.cc:482] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:00.592908 18311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57276 (request call id 581) took 1299 ms. Trace:
I20260807 03:39:00.593264 18311 rpcz_store.cc:276] 0807 03:38:59.293024 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:59.293179 (+ 155us) service_pool.cc:227] Handling call
0807 03:38:59.462629 (+169450us) spinlock_profiling.cc:243] Waited 2.35 ms on lock 0x7b58000e0498. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f25f851 00007fe98f25f702 00007fe99814fb09 00007fe99708b730 00007fe99715179b 00007fe997151821 00007fe9971152b0 00007fe9971007f9 00007fe9971002a2 00007fe9907e6684
0807 03:38:59.462820 (+ 191us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:38:59.555676 (+ 92856us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:38:59.555722 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:38:59.556270 (+ 548us) log.cc:842] Serialized 26 byte log entry
0807 03:38:59.795033 (+238763us) raft_consensus.cc:1631] Marking committed up to 1
0807 03:38:59.795089 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:00.592575 (+797486us) spinlock_profiling.cc:243] Waited 69.3 ms on lock 0x7b54002b34d4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:39:00.592628 (+ 53us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:00.592680 (+ 52us) raft_consensus.cc:1671] finished
0807 03:39:00.592691 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:00.592813 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16282,"raft.queue_time_us":26549,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":6884,"spinlock_wait_cycles":164846336,"thread_start_us":133213,"threads_started":1,"wal-append.queue_time_us":563666}
W20260807 03:39:00.594791 20078 consensus_meta.cc:287] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 1.057s user 0.010s sys 0.000s
I20260807 03:39:00.595467 18311 raft_consensus.cc:1226] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260807 03:39:00.595175 20078 raft_consensus.cc:504] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:00.592504 20152 raft_consensus.cc:504] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:00.598181 20078 leader_election.cc:288] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:00.598596 20152 leader_election.cc:288] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:00.547891 20177 raft_consensus.cc:504] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
W20260807 03:39:00.541555 18138 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.004s user 0.007s sys 0.020s
I20260807 03:39:00.582134 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:00.565481 20169 raft_consensus.cc:504] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:00.601861 19847 raft_consensus.cc:482] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:00.711634 19847 raft_consensus.cc:504] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:00.706070 20124 raft_consensus.cc:482] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:00.712680 20124 raft_consensus.cc:504] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:00.713749 19847 leader_election.cc:288] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:00.723448 18311 raft_consensus.cc:1226] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20260807 03:39:00.707515 18138 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.179s user 0.007s sys 0.020s
I20260807 03:39:00.714494 20062 leader_election.cc:288] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:00.709053 19874 raft_consensus.cc:482] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:00.757586 19874 raft_consensus.cc:504] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:00.732142 20169 leader_election.cc:288] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:00.708736 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:00.761217 18184 raft_consensus.cc:2379] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5c9b5062bdfa46af9396f1a85025a331 in current term 4: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:00.761410 19874 leader_election.cc:288] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:00.747875 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:00.762485 19240 leader_election.cc:302] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:00.762638 18075 raft_consensus.cc:2454] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 1.
I20260807 03:39:00.763365 20152 raft_consensus.cc:3037] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 3 FOLLOWER]: Advancing to term 4
W20260807 03:39:00.751036 20068 consensus_meta.cc:287] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 1.089s user 0.008s sys 0.004s
I20260807 03:39:00.752560 20144 leader_election.cc:288] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:00.763727 19239 leader_election.cc:302] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:00.753384 19471 raft_consensus.cc:3037] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:00.754719 20173 raft_consensus.cc:482] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
W20260807 03:39:00.756722 20034 consensus_meta.cc:287] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 1.089s user 0.008s sys 0.003s
I20260807 03:39:00.757084 20124 leader_election.cc:288] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:00.779639 18191 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.699s user 0.065s sys 0.117s
W20260807 03:39:00.785192 18191 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.704s user 0.065s sys 0.117s
I20260807 03:39:00.788437 19945 raft_consensus.cc:482] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:00.793599 19945 raft_consensus.cc:504] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:00.789506 20034 raft_consensus.cc:482] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:00.799439 20136 raft_consensus.cc:482] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:00.799218 20034 raft_consensus.cc:504] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:00.800022 18069 raft_consensus.cc:2396] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 03:39:00.784373 20173 raft_consensus.cc:504] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:00.811430 18305 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.689s user 0.016s sys 0.015s
W20260807 03:39:00.829778 18305 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.707s user 0.016s sys 0.015s
W20260807 03:39:00.810930 18141 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.574s user 0.009s sys 0.009s
W20260807 03:39:00.832137 18141 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.596s user 0.010s sys 0.009s
W20260807 03:39:00.832643 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 599267us
I20260807 03:39:00.835970 20034 leader_election.cc:288] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:00.836373 20156 raft_consensus.cc:482] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:00.836843 19847 raft_consensus.cc:482] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
W20260807 03:39:00.842545 18249 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.521s user 0.008s sys 0.022s
W20260807 03:39:00.811271 19917 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.520s user 0.003s sys 0.016s
W20260807 03:39:00.845729 18249 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.524s user 0.008s sys 0.022s
I20260807 03:39:00.810727 20159 raft_consensus.cc:482] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:00.810639 20062 raft_consensus.cc:482] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 03:39:00.848395 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 441974us
I20260807 03:39:00.833669 20091 raft_consensus.cc:482] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
W20260807 03:39:00.850836 18309 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.010s sys 0.022s
I20260807 03:39:00.799988 20136 raft_consensus.cc:504] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:00.852712 18309 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.519s user 0.011s sys 0.022s
I20260807 03:39:00.855891 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:00.848657 18140 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.722s user 0.004s sys 0.016s
I20260807 03:39:00.858001 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:00.860605 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:00.849363 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 527916us
W20260807 03:39:00.858332 18140 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.734s user 0.004s sys 0.017s
W20260807 03:39:00.862931 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 738359us
I20260807 03:39:00.902535 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:00.903429 18300 raft_consensus.cc:2396] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 03:39:00.904286 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:00.904515 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ca0223f47b46fd955704eba5259370" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:00.905087 18075 raft_consensus.cc:2454] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:39:00.846537 20034 raft_consensus.cc:482] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:00.851462 20144 raft_consensus.cc:482] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:00.910745 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:00.911981 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 49343.7us
I20260807 03:39:00.913662 18305 raft_consensus.cc:1261] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260807 03:39:00.847078 19917 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.570s user 0.003s sys 0.016s
I20260807 03:39:00.846423 20156 raft_consensus.cc:504] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:00.917407 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:00.921808 18311 raft_consensus.cc:2454] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 5.
I20260807 03:39:00.917138 18249 leader_election.cc:302] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:00.920063 18300 raft_consensus.cc:1261] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:39:00.847621 19847 raft_consensus.cc:504] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:00.926080 18249 leader_election.cc:302] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:00.909685 20177 leader_election.cc:288] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:00.928390 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:00.923578 18027 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.15.198.130:37057 timed out after 1.693s (SENT) [suppressed 5 similar messages]
I20260807 03:39:00.911422 20126 raft_consensus.cc:482] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
W20260807 03:39:00.921166 20191 log.cc:924] Time spent T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.132s user 0.000s sys 0.001s
I20260807 03:39:00.936760 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:00.853075 20062 raft_consensus.cc:504] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:00.856330 20068 consensus_queue.cc:1046] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.068s
I20260807 03:39:00.863292 19874 consensus_queue.cc:1035] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.063s
I20260807 03:39:00.855500 20091 raft_consensus.cc:504] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:00.940940 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:00.934508 20135 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.841s user 0.001s sys 0.043s
I20260807 03:39:00.923571 19945 leader_election.cc:288] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:00.933208 19825 raft_consensus.cc:2781] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:39:00.853250 20159 raft_consensus.cc:504] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:00.932920 19743 raft_consensus.cc:2781] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:39:00.949832 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:00.940152 20018 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.454s user 0.022s sys 0.059s
I20260807 03:39:00.921213 20144 raft_consensus.cc:504] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:00.957638 18311 raft_consensus.cc:2454] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 4.
I20260807 03:39:00.962245 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:00.954726 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:00.933089 20169 raft_consensus.cc:482] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:00.919447 20034 raft_consensus.cc:504] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:00.937623 20177 raft_consensus.cc:482] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
W20260807 03:39:00.948030 20135 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.854s user 0.002s sys 0.043s
W20260807 03:39:00.950205 19480 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.724s user 0.017s sys 0.034s
I20260807 03:39:01.001433 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:00.957455 20018 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.472s user 0.022s sys 0.059s
I20260807 03:39:00.996850 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:01.004698 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:01.020272 18193 raft_consensus.cc:2454] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:00.956316 19291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
W20260807 03:39:01.004297 19480 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.783s user 0.017s sys 0.034s
I20260807 03:39:01.001458 19743 raft_consensus.cc:686] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:00.938592 20126 raft_consensus.cc:504] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:01.054059 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:01.064749 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:01.100718 18070 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33254 (request call id 904) took 1823 ms. Trace:
I20260807 03:39:01.171316 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:01.175757 18190 raft_consensus.cc:2454] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:39:01.003468 20068 raft_consensus.cc:3032] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:39:01.178432 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:01.053282 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:00.992511 19239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 228940us
I20260807 03:39:01.176855 18070 rpcz_store.cc:276] 0807 03:38:59.277403 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:59.279922 (+ 2519us) service_pool.cc:227] Handling call
0807 03:38:59.294523 (+ 14601us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:38:59.294909 (+ 386us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:38:59.294949 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:38:59.309098 (+ 14149us) log.cc:842] Serialized 26 byte log entry
0807 03:38:59.534802 (+225704us) raft_consensus.cc:1631] Marking committed up to 2
0807 03:38:59.603375 (+ 68573us) log.cc:842] Serialized 14 byte log entry
0807 03:39:00.597653 (+994278us) spinlock_profiling.cc:243] Waited 38.9 ms on lock 0x7b18003a99c0. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe9981368b1 00007fe998143661 00007fe98f297638 00007fe98f29ab11 00007fe98f29aac9 00007fe98f29aa59 00007fe98f29aa21 00007fe98f299d1d 00007fe9984fb204 00007fe9984fb039 00007fe98f279dc6
0807 03:39:01.100243 (+502590us) spinlock_profiling.cc:243] Waited 41.2 ms on lock 0x7b18003a99c0. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe9981368b1 00007fe998143661 00007fe991e4e82c 00007fe98f2799dc 00007fe9980b0a39 00007fe98f2968bb 00007fe98f294cbf 00007fe98f294155 00007fe9921b73d0 00007fe9921c9576 00007fe9921da0d7
0807 03:39:01.100389 (+ 146us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:01.100471 (+ 82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:01.100509 (+ 38us) raft_consensus.cc:1671] finished
0807 03:39:01.100523 (+ 14us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:01.100650 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31911,"prepare.queue_time_us":241513,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":30348,"raft.queue_time_us":235576,"raft.run_cpu_time_us":259,"raft.run_wall_time_us":57714,"spinlock_wait_cycles":184326400,"thread_start_us":299993,"threads_started":3,"wal-append.queue_time_us":440634}
I20260807 03:39:01.196523 18180 raft_consensus.cc:2379] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 2: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
W20260807 03:39:01.195968 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 272402us
I20260807 03:39:01.193261 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:01.142554 20210 raft_consensus.cc:482] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:01.039634 20173 leader_election.cc:288] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:01.040133 20052 raft_consensus.cc:482] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:01.217165 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:01.051553 19825 raft_consensus.cc:686] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 2 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:01.038125 18297 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57268 (request call id 853) took 1456 ms. Trace:
I20260807 03:39:01.151320 20152 raft_consensus.cc:2726] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:01.051347 20124 raft_consensus.cc:482] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:01.067243 19874 raft_consensus.cc:482] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:01.142230 20207 raft_consensus.cc:482] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:01.178156 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:01.077499 20179 raft_consensus.cc:482] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:01.018772 20169 raft_consensus.cc:504] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:01.178009 20073 raft_consensus.cc:482] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:01.187079 20072 raft_consensus.cc:482] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:01.190917 20068 raft_consensus.cc:729] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:01.237921 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:01.256034 18305 raft_consensus.cc:2396] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20260807 03:39:01.221108 20135 raft_consensus.cc:482] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:01.201320 19480 raft_consensus.cc:482] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 03:39:01.232650 20184 log.cc:924] Time spent T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.118s user 0.000s sys 0.001s
I20260807 03:39:01.055805 20177 raft_consensus.cc:504] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:01.231714 20151 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.664s user 0.015s sys 0.012s
I20260807 03:39:01.268101 20135 raft_consensus.cc:504] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:01.271072 19480 raft_consensus.cc:504] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:01.255930 20207 raft_consensus.cc:504] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:01.263597 20146 raft_consensus.cc:482] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:01.258988 18298 raft_consensus.cc:2396] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260807 03:39:01.219776 19945 raft_consensus.cc:482] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:01.225831 20052 raft_consensus.cc:504] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:01.262998 20206 raft_consensus.cc:482] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:01.225018 20173 raft_consensus.cc:482] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:01.225031 20156 leader_election.cc:288] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:01.266824 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:01.286049 18305 raft_consensus.cc:2396] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 03:39:01.224675 20136 leader_election.cc:288] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:01.282382 20146 raft_consensus.cc:504] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:01.285173 20177 leader_election.cc:288] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:01.282332 19945 raft_consensus.cc:504] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:01.288005 20146 leader_election.cc:288] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:01.288551 20212 log.cc:924] Time spent T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.169s user 0.001s sys 0.000s
I20260807 03:39:01.288892 19945 leader_election.cc:288] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:01.253830 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:01.248783 20152 raft_consensus.cc:482] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:01.247789 20040 raft_consensus.cc:482] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:01.257313 20091 leader_election.cc:288] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:01.243659 20159 leader_election.cc:288] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:01.275748 19480 leader_election.cc:288] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:01.270954 20183 log.cc:924] Time spent T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20260807 03:39:01.276978 18189 raft_consensus.cc:2379] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 3: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
W20260807 03:39:01.271945 20151 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.998s user 0.015s sys 0.013s
I20260807 03:39:01.267581 19743 consensus_queue.cc:237] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:01.259939 19847 leader_election.cc:288] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:01.246908 19874 raft_consensus.cc:504] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:01.298821 20146 raft_consensus.cc:482] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 03:39:01.307510 20197 log.cc:924] Time spent T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.324s user 0.000s sys 0.001s
I20260807 03:39:01.259487 20073 raft_consensus.cc:504] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:01.281013 20077 raft_consensus.cc:482] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:01.259992 20124 raft_consensus.cc:504] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:01.303304 20126 leader_election.cc:288] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:01.304124 20229 raft_consensus.cc:482] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:01.264241 19825 consensus_queue.cc:237] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:01.290853 20144 leader_election.cc:288] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:01.306644 19480 raft_consensus.cc:482] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:01.275610 20135 leader_election.cc:288] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:01.259044 20179 raft_consensus.cc:504] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:01.289693 20222 raft_consensus.cc:2781] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:39:01.297240 20052 leader_election.cc:288] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:01.219977 20210 raft_consensus.cc:504] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:01.298149 20206 raft_consensus.cc:504] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:01.329073 20169 leader_election.cc:288] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:01.329808 20073 leader_election.cc:288] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:01.330735 19871 raft_consensus.cc:3032] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:39:01.305061 20034 leader_election.cc:288] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:01.330958 18073 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33254 (request call id 911) took 2048 ms. Trace:
I20260807 03:39:01.260596 20072 raft_consensus.cc:504] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:01.291834 20173 raft_consensus.cc:504] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:01.324215 20146 raft_consensus.cc:504] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:01.221657 18297 rpcz_store.cc:276] 0807 03:38:59.565947 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:59.571924 (+ 5977us) service_pool.cc:227] Handling call
0807 03:38:59.574268 (+ 2344us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:38:59.574602 (+ 334us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:38:59.574641 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:38:59.575097 (+ 456us) log.cc:842] Serialized 26 byte log entry
0807 03:39:00.530238 (+955141us) raft_consensus.cc:1631] Marking committed up to 2
0807 03:39:00.946259 (+416021us) log.cc:842] Serialized 14 byte log entry
0807 03:39:00.958645 (+ 12386us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:00.982619 (+ 23974us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:00.996780 (+ 14161us) raft_consensus.cc:1671] finished
0807 03:39:00.996797 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:01.022008 (+ 25211us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12427,"prepare.queue_time_us":603799,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":6290,"raft.run_cpu_time_us":286,"raft.run_wall_time_us":28370,"thread_start_us":1337310,"threads_started":2,"wal-append.queue_time_us":968996}
I20260807 03:39:01.300994 20152 raft_consensus.cc:504] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:01.294415 20068 consensus_queue.cc:260] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:01.350756 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:01.308434 20040 raft_consensus.cc:504] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:01.329854 18310 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57268 (request call id 857) took 1752 ms. Trace:
W20260807 03:39:01.370772 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 331583us
I20260807 03:39:01.388466 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:01.330305 20077 raft_consensus.cc:504] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:01.377213 18073 rpcz_store.cc:276] 0807 03:38:59.282635 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:59.322959 (+ 40324us) service_pool.cc:227] Handling call
0807 03:38:59.368799 (+ 45840us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:38:59.484492 (+115693us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:38:59.484520 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:38:59.484967 (+ 447us) log.cc:842] Serialized 26 byte log entry
0807 03:39:00.577832 (+1092865us) raft_consensus.cc:1631] Marking committed up to 2
0807 03:39:00.617017 (+ 39185us) log.cc:842] Serialized 14 byte log entry
0807 03:39:00.617294 (+ 277us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:00.617354 (+ 60us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:01.330736 (+713382us) raft_consensus.cc:1671] finished
0807 03:39:01.330762 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:01.330891 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":126785,"prepare.queue_time_us":111242,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":3984,"raft.queue_time_us":144228,"raft.run_cpu_time_us":322,"raft.run_wall_time_us":78644,"thread_start_us":1092484,"threads_started":1,"wal-append.queue_time_us":1315075}
I20260807 03:39:01.359301 20156 raft_consensus.cc:482] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:01.389043 18248 leader_election.cc:302] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:01.402144 18295 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57268 (request call id 868) took 1341 ms. Trace:
I20260807 03:39:01.377558 20052 raft_consensus.cc:482] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:01.372362 19871 raft_consensus.cc:729] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:01.371954 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:01.409651 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:01.361555 20222 raft_consensus.cc:482] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:01.345824 19480 raft_consensus.cc:504] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:01.431849 18190 raft_consensus.cc:2454] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:39:01.441005 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:01.402559 18295 rpcz_store.cc:276] 0807 03:39:00.060438 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:00.071645 (+ 11207us) service_pool.cc:227] Handling call
0807 03:39:00.071938 (+ 293us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:00.099882 (+ 27944us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:00.099912 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:39:00.130550 (+ 30638us) log.cc:842] Serialized 26 byte log entry
0807 03:39:00.786258 (+655708us) raft_consensus.cc:1631] Marking committed up to 2
0807 03:39:00.998298 (+212040us) log.cc:842] Serialized 14 byte log entry
0807 03:39:01.020163 (+ 21865us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:01.020261 (+ 98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:01.401969 (+381708us) raft_consensus.cc:1671] finished
0807 03:39:01.401986 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:01.402096 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":103611,"prepare.queue_time_us":421945,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":57,"raft.queue_time_us":75194,"raft.run_cpu_time_us":288,"raft.run_wall_time_us":51041,"thread_start_us":683218,"threads_started":2,"wal-append.queue_time_us":687955}
I20260807 03:39:01.434295 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:01.443399 18193 raft_consensus.cc:2379] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 5: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:01.409950 20068 raft_consensus.cc:3037] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:01.408532 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:01.450559 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:01.451197 18190 raft_consensus.cc:2454] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:39:01.452311 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:01.432801 20073 raft_consensus.cc:2781] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260807 03:39:01.449546 20141 log.cc:924] Time spent T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 1.681s user 0.005s sys 0.017s
I20260807 03:39:01.454118 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:01.346288 20229 raft_consensus.cc:504] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:01.456127 18193 raft_consensus.cc:2396] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 03:39:01.457949 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ca0223f47b46fd955704eba5259370" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:01.408619 20156 raft_consensus.cc:504] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:01.445312 20018 raft_consensus.cc:3032] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:39:01.438750 20222 raft_consensus.cc:3037] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:01.464776 18140 leader_election.cc:302] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:01.440423 20091 consensus_queue.cc:1035] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.022s
I20260807 03:39:01.432767 20052 raft_consensus.cc:504] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:01.505055 18191 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44662 (request call id 561) took 1832 ms. Trace:
I20260807 03:39:01.456374 18184 raft_consensus.cc:2454] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 4.
I20260807 03:39:01.460708 20243 raft_consensus.cc:482] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:01.462467 20234 raft_consensus.cc:482] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:01.477415 20124 leader_election.cc:288] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:01.457526 20073 raft_consensus.cc:482] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
W20260807 03:39:01.466674 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 381977us
I20260807 03:39:01.568540 18027 leader_election.cc:302] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:01.449622 19871 consensus_queue.cc:260] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:01.466475 20018 raft_consensus.cc:729] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:01.574627 20159 raft_consensus.cc:2781] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:01.575052 18027 leader_election.cc:302] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:01.575212 20072 leader_election.cc:288] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
W20260807 03:39:01.547221 19240 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 91943.8us
I20260807 03:39:01.576330 20073 raft_consensus.cc:3037] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:01.541723 20034 raft_consensus.cc:482] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:01.579075 18249 leader_election.cc:302] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:01.602458 18249 leader_election.cc:302] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:01.581149 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:01.597332 19291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:01.576992 19743 raft_consensus.cc:482] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:01.584450 20238 raft_consensus.cc:482] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:01.621238 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:01.585428 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:01.600180 20251 log.cc:924] Time spent T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20260807 03:39:01.561412 20196 raft_consensus.cc:482] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:01.567058 20135 raft_consensus.cc:482] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:01.591619 18081 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33268 (request call id 825) took 2475 ms. Trace:
I20260807 03:39:01.560532 20244 consensus_queue.cc:1046] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.053s
I20260807 03:39:01.545331 19874 leader_election.cc:288] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:01.582134 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:01.600751 20173 leader_election.cc:288] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:01.618703 18190 raft_consensus.cc:1261] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 03:39:01.576427 20228 raft_consensus.cc:482] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:01.608014 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:01.639299 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:01.596684 19917 raft_consensus.cc:2726] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:01.591454 20159 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:01.626346 20204 raft_consensus.cc:2726] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:01.400568 18310 rpcz_store.cc:276] 0807 03:38:59.577240 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:59.582606 (+ 5366us) service_pool.cc:227] Handling call
0807 03:38:59.656899 (+ 74293us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:38:59.793763 (+136864us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:38:59.810140 (+ 16377us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:38:59.951121 (+140981us) log.cc:842] Serialized 26 byte log entry
0807 03:39:00.381441 (+430320us) raft_consensus.cc:1631] Marking committed up to 1
0807 03:39:00.405199 (+ 23758us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:01.329479 (+924280us) spinlock_profiling.cc:243] Waited 12.3 ms on lock 0x7b54002b39d4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:39:01.329524 (+ 45us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:01.329571 (+ 47us) raft_consensus.cc:1671] finished
0807 03:39:01.329584 (+ 13us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:01.329797 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3218,"raft.queue_time_us":18434,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":142624,"spinlock_wait_cycles":28229376,"thread_start_us":377300,"threads_started":1,"wal-append.queue_time_us":607001}
I20260807 03:39:01.621907 18080 raft_consensus.cc:2454] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 1.
I20260807 03:39:01.541669 18191 rpcz_store.cc:276] 0807 03:38:59.646959 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:59.652167 (+ 5208us) service_pool.cc:227] Handling call
0807 03:38:59.652599 (+ 432us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:00.028394 (+375795us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:00.028423 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:39:00.029155 (+ 732us) log.cc:842] Serialized 26 byte log entry
0807 03:39:00.785602 (+756447us) raft_consensus.cc:1631] Marking committed up to 1
0807 03:39:00.785638 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:01.465265 (+679627us) spinlock_profiling.cc:243] Waited 170 ms on lock 0x7b54002cf1d4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:39:01.465329 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:01.465388 (+ 59us) raft_consensus.cc:1671] finished
0807 03:39:01.465406 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:01.478977 (+ 13571us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":206045,"raft.queue_time_us":208855,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":9156,"spinlock_wait_cycles":392042880,"thread_start_us":704467,"threads_started":1,"wal-append.queue_time_us":728114}
I20260807 03:39:01.591454 20124 raft_consensus.cc:482] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:01.560719 20236 raft_consensus.cc:482] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:01.602949 19871 raft_consensus.cc:3037] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
W20260807 03:39:01.466065 18138 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.132:41381, user_credentials={real_user=slave}} blocked reactor thread for 279049us
I20260807 03:39:01.577262 20177 consensus_queue.cc:1046] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20260807 03:39:01.606204 20126 raft_consensus.cc:2726] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:01.598778 20235 raft_consensus.cc:482] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:01.604857 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:01.630499 18081 rpcz_store.cc:276] 0807 03:38:59.105478 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:59.113550 (+ 8072us) service_pool.cc:227] Handling call
0807 03:38:59.118707 (+ 5157us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:38:59.119141 (+ 434us) raft_consensus.cc:1520] Early marking committed up to index 1
0807 03:38:59.119180 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:38:59.142689 (+ 23509us) log.cc:842] Serialized 26 byte log entry
0807 03:38:59.245822 (+103133us) spinlock_profiling.cc:243] Waited 15.9 ms on lock 0x7b18000d0c00. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe9981368b1 00007fe998143661 00007fe98f279345 00007fe9980b0a39 00007fe98f2968bb 00007fe98f294cbf 00007fe98f296e22 00007fe991e405bb 00007fe991e46063 00007fe991e46411 00007fe9921935cf
0807 03:38:59.528753 (+282931us) raft_consensus.cc:1631] Marking committed up to 1
0807 03:38:59.528784 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:38:59.528845 (+ 61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:01.307131 (+1778286us) spinlock_profiling.cc:243] Waited 324 ms on lock 0x7b44000232cc. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f1a9a95 00007fe98f1aa4b2 00007fe991e52511 00007fe991e4de1a 00007fe991e41bee 00007fe991e40f36 00007fe991e40b58 00007fe991e4a551 00007fe991e4a509 00007fe991e4a499
0807 03:39:01.580629 (+273498us) raft_consensus.cc:1671] finished
0807 03:39:01.580649 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:01.580773 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10620,"raft.queue_time_us":112,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":7479,"spinlock_wait_cycles":782003840,"thread_start_us":280633,"threads_started":1,"wal-append.queue_time_us":410038}
I20260807 03:39:01.606561 20072 raft_consensus.cc:2781] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:39:01.645772 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:01.686932 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:01.624902 20152 leader_election.cc:288] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:01.627568 20146 leader_election.cc:288] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:01.649864 20250 raft_consensus.cc:482] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:01.637288 20135 raft_consensus.cc:504] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:01.562516 20243 raft_consensus.cc:504] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:01.670935 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:01.671947 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:01.671206 19239 leader_election.cc:302] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:01.674327 19297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:01.646873 20159 raft_consensus.cc:3037] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:01.640264 20040 leader_election.cc:288] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:01.641098 19874 raft_consensus.cc:482] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:01.644766 20173 raft_consensus.cc:482] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:01.628022 20034 raft_consensus.cc:504] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:01.685719 20229 leader_election.cc:288] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:01.628923 19743 raft_consensus.cc:504] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:01.687572 18070 raft_consensus.cc:2454] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
W20260807 03:39:01.692188 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 112399us
I20260807 03:39:01.665202 20228 raft_consensus.cc:504] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:01.710712 20228 leader_election.cc:288] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:01.702997 19297 raft_consensus.cc:2454] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:39:01.691004 20072 raft_consensus.cc:482] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:01.694762 19480 leader_election.cc:288] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:01.661324 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:01.670285 20179 leader_election.cc:288] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:01.676029 19917 raft_consensus.cc:482] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:01.670506 20206 leader_election.cc:288] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:01.702111 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:01.705119 20220 log.cc:924] Time spent T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.239s user 0.000s sys 0.001s
I20260807 03:39:01.705116 20177 raft_consensus.cc:482] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:01.717758 18025 leader_election.cc:302] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:01.680085 20236 raft_consensus.cc:504] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:01.567540 20234 raft_consensus.cc:504] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:01.706907 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 242207us
I20260807 03:39:01.723688 18081 raft_consensus.cc:2454] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:39:01.723636 18140 leader_election.cc:302] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:01.695029 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:01.712824 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:01.675640 20124 raft_consensus.cc:504] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:01.696858 20210 leader_election.cc:288] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:01.695994 18071 raft_consensus.cc:2454] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:39:01.701098 20156 leader_election.cc:288] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:01.711501 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:01.717741 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:01.708214 20052 leader_election.cc:288] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:01.714221 19874 raft_consensus.cc:504] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:01.660774 20196 raft_consensus.cc:504] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:01.640928 20238 raft_consensus.cc:504] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:01.706892 20250 raft_consensus.cc:504] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:01.699980 20235 raft_consensus.cc:504] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:01.591645 20151 consensus_queue.cc:1159] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "66a36e1ac8e945e893668e282d394c49" responder_term: 4 status { last_received { term: 1 index: 1 } last_committed_idx: 1 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 3. Current term is 4. Ops: [3.2-3.2]" } } last_received_current_leader { term: 0 index: 0 } }
I20260807 03:39:01.714049 20173 raft_consensus.cc:504] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:01.720129 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:01.717185 20263 consensus_queue.cc:1035] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.145s
I20260807 03:39:01.806741 20204 raft_consensus.cc:482] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:01.785063 20175 raft_consensus.cc:482] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:01.806540 20075 raft_consensus.cc:482] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
W20260807 03:39:01.787164 20221 log.cc:924] Time spent T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.232s user 0.000s sys 0.003s
I20260807 03:39:01.855252 20196 leader_election.cc:288] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:01.855759 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:01.907333 18070 raft_consensus.cc:2454] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:39:01.722052 19832 raft_consensus.cc:3032] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:39:01.856340 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:01.721247 19917 raft_consensus.cc:504] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:01.749506 18140 leader_election.cc:302] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:01.856675 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:01.809068 18249 leader_election.cc:302] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:01.714082 20254 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:01.801234 20258 raft_consensus.cc:482] T af8877c3fbba4ab18765832f1390921f P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:01.858191 18080 raft_consensus.cc:2454] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:39:01.779125 20135 leader_election.cc:288] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:01.760703 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:01.716856 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:01.724825 18191 raft_consensus.cc:2454] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:39:01.924824 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:01.925536 18191 raft_consensus.cc:2454] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:39:01.779516 20172 raft_consensus.cc:2781] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:39:01.738423 18193 raft_consensus.cc:1261] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:39:01.735060 18075 raft_consensus.cc:2454] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:39:01.927356 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:01.715116 20072 raft_consensus.cc:3037] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:01.860773 20250 leader_election.cc:288] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:01.747423 18183 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44662 (request call id 546) took 2977 ms (client timeout 1693 ms). Trace:
I20260807 03:39:01.716244 20152 raft_consensus.cc:2781] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:39:01.855270 20238 leader_election.cc:288] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:01.731266 20206 raft_consensus.cc:2781] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:39:01.859120 19874 leader_election.cc:288] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:01.777943 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:01.722695 20177 raft_consensus.cc:504] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:01.936465 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:01.864437 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 153538us
I20260807 03:39:01.864820 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:01.810256 20040 raft_consensus.cc:482] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
W20260807 03:39:01.875805 20242 log.cc:924] Time spent T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.109s user 0.000s sys 0.001s
I20260807 03:39:01.923267 20196 raft_consensus.cc:2726] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:01.893810 20243 leader_election.cc:288] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
W20260807 03:39:01.899114 20264 log.cc:924] Time spent T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.110s user 0.001s sys 0.000s
I20260807 03:39:01.900032 19743 leader_election.cc:288] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:01.903903 20169 raft_consensus.cc:482] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:01.905719 20234 leader_election.cc:288] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:01.908991 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:01.902788 18248 leader_election.cc:302] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:01.912611 19924 leader_election.cc:288] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:01.913182 20161 raft_consensus.cc:3032] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:39:01.910511 19832 raft_consensus.cc:729] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:01.917052 18071 raft_consensus.cc:2454] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
W20260807 03:39:01.920411 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 111447us
W20260807 03:39:01.919763 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 170338us
I20260807 03:39:01.945178 20205 raft_consensus.cc:2781] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:01.923161 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:01.930984 20269 log.cc:924] Time spent T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.129s user 0.000s sys 0.000s
I20260807 03:39:01.931059 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:01.933514 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:01.908404 20034 leader_election.cc:288] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:01.966847 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:01.967568 18071 raft_consensus.cc:2454] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:39:01.938315 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:01.858949 20124 leader_election.cc:288] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:01.908581 20075 raft_consensus.cc:504] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:01.973457 20075 leader_election.cc:288] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:01.945943 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:01.974109 19292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:01.974750 20173 leader_election.cc:288] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:01.933524 20206 raft_consensus.cc:482] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
W20260807 03:39:01.948036 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 45357.3us
I20260807 03:39:01.978284 18248 leader_election.cc:302] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:01.934314 20235 leader_election.cc:288] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:01.939209 20172 raft_consensus.cc:686] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:01.934937 18310 raft_consensus.cc:2454] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 2.
I20260807 03:39:01.931464 20236 leader_election.cc:288] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
W20260807 03:39:01.931871 18183 rpcz_store.cc:269] 0807 03:38:58.761570 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:58.763935 (+ 2365us) service_pool.cc:227] Handling call
0807 03:38:59.095709 (+331774us) spinlock_profiling.cc:243] Waited 1.36 ms on lock 0x7b5800003998. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f25f851 00007fe98f25f702 00007fe99814fb09 00007fe99708b730 00007fe99715179b 00007fe997151821 00007fe9971011b0 00007fe997100a94 00007fe9907e8c04 00007fe9907e8b92
0807 03:39:01.722883 (+2627174us) spinlock_profiling.cc:243] Waited 171 ms on lock 0x7b5400172d54. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:01.730762 (+ 7879us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":395752576}
I20260807 03:39:01.954756 20161 raft_consensus.cc:729] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:01.917472 20144 raft_consensus.cc:482] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:01.958009 18140 leader_election.cc:302] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:01.954579 20169 raft_consensus.cc:504] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:01.934696 20258 raft_consensus.cc:504] T af8877c3fbba4ab18765832f1390921f P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:01.961819 18070 raft_consensus.cc:2454] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
I20260807 03:39:01.966192 20205 raft_consensus.cc:482] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:01.918583 20204 raft_consensus.cc:504] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:01.971444 20213 log.cc:924] Time spent T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.798s user 0.002s sys 0.000s
I20260807 03:39:02.022841 20247 raft_consensus.cc:2781] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:39:02.023371 20247 raft_consensus.cc:482] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:02.023730 20247 raft_consensus.cc:3037] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:01.975339 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:01.976459 18249 leader_election.cc:302] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:01.976871 19295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:01.977486 20206 raft_consensus.cc:3037] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:01.943221 20040 raft_consensus.cc:504] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:01.980969 20172 consensus_queue.cc:237] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:01.922219 20254 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:01.928267 20175 raft_consensus.cc:504] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:01.956401 18190 raft_consensus.cc:1261] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 03:39:01.952356 20222 raft_consensus.cc:504] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:02.009631 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:02.010754 19917 leader_election.cc:288] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:02.024596 18297 raft_consensus.cc:2454] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3.
I20260807 03:39:02.103868 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:02.115638 18312 raft_consensus.cc:3037] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:02.036798 18187 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44660 (request call id 993) took 1091 ms. Trace:
I20260807 03:39:02.116489 18295 raft_consensus.cc:2454] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 2.
I20260807 03:39:02.116868 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:02.105432 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 147469us
I20260807 03:39:02.117157 18248 leader_election.cc:302] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:02.033507 18311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57276 (request call id 600) took 1105 ms. Trace:
I20260807 03:39:01.985713 19924 raft_consensus.cc:2781] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:39:02.116658 18187 rpcz_store.cc:276] 0807 03:39:00.945871 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:00.948769 (+ 2898us) service_pool.cc:227] Handling call
0807 03:39:02.036137 (+1087368us) spinlock_profiling.cc:243] Waited 4.03 ms on lock 0x7b5800003998. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f25f851 00007fe98f25f702 00007fe99814fb09 00007fe99708b730 00007fe99715179b 00007fe997151821 00007fe9971152b0 00007fe9971007f9 00007fe9971002a2 00007fe9907e6684
0807 03:39:02.036329 (+ 192us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:02.036501 (+ 172us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:02.036527 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:02.036567 (+ 40us) raft_consensus.cc:1631] Marking committed up to 1
0807 03:39:02.036595 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:02.036644 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:02.036743 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9258368}
I20260807 03:39:02.121112 18027 leader_election.cc:302] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
W20260807 03:39:02.119415 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 143030us
I20260807 03:39:02.118744 18140 leader_election.cc:302] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:02.120317 18311 rpcz_store.cc:276] 0807 03:39:00.918988 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:00.925329 (+ 6341us) service_pool.cc:227] Handling call
0807 03:39:02.024684 (+1099355us) spinlock_profiling.cc:243] Waited 16.1 ms on lock 0x7b540035a054. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:02.024794 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":37031680}
I20260807 03:39:02.122890 18027 leader_election.cc:302] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:01.983115 19743 raft_consensus.cc:2781] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:39:02.126799 18027 leader_election.cc:302] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:02.135917 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:02.137073 18081 raft_consensus.cc:1261] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 03:39:02.132332 18140 leader_election.cc:302] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:02.127964 18248 leader_election.cc:302] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:01.983072 20034 raft_consensus.cc:482] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:01.986195 20173 raft_consensus.cc:482] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:02.103458 20042 raft_consensus.cc:482] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:02.167703 18297 raft_consensus.cc:2454] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3.
I20260807 03:39:01.989025 20177 leader_election.cc:288] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:02.143095 18027 leader_election.cc:302] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:01.987532 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:02.024827 20205 raft_consensus.cc:3037] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:02.169788 18140 leader_election.cc:302] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:02.139714 18249 leader_election.cc:302] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:02.126582 19945 raft_consensus.cc:2781] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election won for term 3
W20260807 03:39:02.173573 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 45370.4us
I20260807 03:39:02.181139 18248 leader_election.cc:302] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:02.166631 20075 consensus_queue.cc:1035] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.010s
I20260807 03:39:02.128773 19917 raft_consensus.cc:2726] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:02.176918 20204 leader_election.cc:288] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:02.177120 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:02.177977 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:02.120958 19924 raft_consensus.cc:482] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:02.168761 20250 raft_consensus.cc:2781] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:39:02.184670 18140 leader_election.cc:302] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
W20260807 03:39:02.186751 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 47138.8us
I20260807 03:39:02.165642 20167 raft_consensus.cc:2781] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:02.024379 20144 raft_consensus.cc:504] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:01.991353 20159 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:02.121843 20281 raft_consensus.cc:3037] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:02.128631 19874 raft_consensus.cc:2781] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:39:02.203578 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:02.172137 20234 raft_consensus.cc:2726] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:02.173607 19297 raft_consensus.cc:1226] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [term 2 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 1. Current term is 2. Ops: []
I20260807 03:39:02.191501 19924 raft_consensus.cc:3037] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:02.202366 18027 leader_election.cc:302] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:02.039520 20018 consensus_queue.cc:260] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:02.183072 20177 raft_consensus.cc:2781] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:02.187071 20243 raft_consensus.cc:2781] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:02.201957 18249 leader_election.cc:302] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:02.222790 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:02.223400 18249 leader_election.cc:302] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:01.988013 19832 consensus_queue.cc:260] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:02.190827 18184 raft_consensus.cc:1261] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 03:39:02.191179 20078 raft_consensus.cc:2726] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:02.195899 20214 raft_consensus.cc:2781] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:02.197021 20250 raft_consensus.cc:482] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:02.210340 18027 leader_election.cc:302] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:02.172003 20173 raft_consensus.cc:504] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:02.174258 20042 raft_consensus.cc:504] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:02.195750 19945 raft_consensus.cc:686] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:02.175935 20034 raft_consensus.cc:504] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:02.125836 20161 consensus_queue.cc:260] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:02.224287 18312 raft_consensus.cc:2454] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:39:02.205147 20167 raft_consensus.cc:482] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:02.220687 19871 raft_consensus.cc:2726] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260807 03:39:02.238336 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 57277.1us
I20260807 03:39:02.207633 19917 raft_consensus.cc:2781] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:39:02.242272 18027 leader_election.cc:302] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:02.207201 19874 raft_consensus.cc:482] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:02.249755 18248 leader_election.cc:302] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
W20260807 03:39:02.261646 18312 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57268 (request call id 861) took 2646 ms (client timeout 1542 ms). Trace:
W20260807 03:39:02.262110 18312 rpcz_store.cc:269] 0807 03:38:59.599912 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:59.601928 (+ 2016us) service_pool.cc:227] Handling call
0807 03:39:02.009443 (+2407515us) spinlock_profiling.cc:243] Waited 7.15 ms on lock 0x7b58000e0498. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f25f851 00007fe98f25f702 00007fe99814fb09 00007fe99708b730 00007fe99715179b 00007fe997151821 00007fe9971011b0 00007fe997100a94 00007fe9907e8c04 00007fe9907e8b92
0807 03:39:02.246563 (+237120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16439040}
I20260807 03:39:02.253383 20258 leader_election.cc:288] T af8877c3fbba4ab18765832f1390921f P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:02.244719 20072 raft_consensus.cc:504] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:02.225131 20157 raft_consensus.cc:2781] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:02.263038 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:02.263742 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:02.222175 20243 raft_consensus.cc:482] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:02.221369 20177 raft_consensus.cc:482] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:02.268389 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:02.243693 20272 raft_consensus.cc:2726] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:02.243647 20250 raft_consensus.cc:3037] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:02.240780 20214 raft_consensus.cc:482] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:02.248416 20018 raft_consensus.cc:3037] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:02.248677 19832 raft_consensus.cc:3037] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:02.239954 20204 raft_consensus.cc:2781] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:02.260447 20167 raft_consensus.cc:3037] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
W20260807 03:39:02.286281 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 46989.3us
I20260807 03:39:02.280067 20169 leader_election.cc:288] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:02.376675 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:02.283401 20177 raft_consensus.cc:3037] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:02.268810 20135 raft_consensus.cc:2781] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260807 03:39:02.396247 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 116303us
I20260807 03:39:02.265144 19874 raft_consensus.cc:3037] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:02.266644 20161 raft_consensus.cc:3037] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:02.281436 18072 raft_consensus.cc:2396] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 03:39:02.266791 20075 raft_consensus.cc:2726] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:02.261521 19917 raft_consensus.cc:482] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
W20260807 03:39:02.353619 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 111771us
I20260807 03:39:02.373571 20204 raft_consensus.cc:482] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:02.363267 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:39:02.284899 20068 consensus_meta.cc:287] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 0.835s user 0.010s sys 0.001s
I20260807 03:39:02.406279 18027 leader_election.cc:302] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:02.279839 20157 raft_consensus.cc:482] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 03:39:02.272295 20073 consensus_meta.cc:287] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.649s user 0.005s sys 0.005s
I20260807 03:39:02.281064 20243 raft_consensus.cc:3037] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:02.375238 20222 leader_election.cc:288] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:02.251713 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:02.393399 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:02.294827 20214 raft_consensus.cc:3037] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:02.429603 20250 raft_consensus.cc:504] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:02.363864 18183 raft_consensus.cc:1226] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 1. Current term is 2. Ops: []
I20260807 03:39:02.431064 20250 leader_election.cc:288] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:02.431592 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:02.365329 20040 leader_election.cc:288] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:02.352624 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:02.416304 18189 raft_consensus.cc:1226] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:02.408411 19295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:02.409060 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:02.400027 20175 leader_election.cc:288] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:02.413177 20169 raft_consensus.cc:2781] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:39:02.414139 18305 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57268 (request call id 880) took 1474 ms. Trace:
I20260807 03:39:02.407683 18027 leader_election.cc:302] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:02.418653 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:02.397598 20135 raft_consensus.cc:482] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:02.294152 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:02.406618 20077 leader_election.cc:288] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:02.420253 20157 raft_consensus.cc:3037] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:02.472477 20144 leader_election.cc:288] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:02.472604 20042 leader_election.cc:288] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:02.435070 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:02.474340 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:39:02.412690 20284 log.cc:924] Time spent T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.122s user 0.001s sys 0.000s
I20260807 03:39:02.487733 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:02.489169 18187 raft_consensus.cc:2454] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:39:02.407089 19917 raft_consensus.cc:3037] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
W20260807 03:39:02.477111 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 69517.8us
I20260807 03:39:02.407302 20204 raft_consensus.cc:3037] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
W20260807 03:39:02.497795 18069 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33254 (request call id 925) took 1930 ms (client timeout 1526 ms). Trace:
I20260807 03:39:02.498739 18027 leader_election.cc:302] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
W20260807 03:39:02.498665 18069 rpcz_store.cc:269] 0807 03:39:00.567508 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:00.576876 (+ 9368us) service_pool.cc:227] Handling call
0807 03:39:02.497596 (+1920720us) spinlock_profiling.cc:243] Waited 194 ms on lock 0x7b5400115dd4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:02.497729 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":445710208}
I20260807 03:39:02.499707 20042 raft_consensus.cc:2726] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:02.500414 18027 leader_election.cc:302] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:02.501314 20042 raft_consensus.cc:2726] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 03:39:02.502852 18027 leader_election.cc:302] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:02.503733 20144 raft_consensus.cc:2726] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:02.483286 20040 consensus_queue.cc:1035] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:39:02.484489 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:02.484584 20247 raft_consensus.cc:504] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:02.486009 20285 log.cc:924] Time spent T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20260807 03:39:02.486889 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:02.506031 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:02.473730 20135 raft_consensus.cc:3037] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:02.421337 20073 raft_consensus.cc:504] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:02.507511 18027 leader_election.cc:302] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:02.471100 20169 raft_consensus.cc:482] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:02.435750 20159 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:02.518092 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:02.519291 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:02.471771 20250 raft_consensus.cc:482] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:02.471406 18305 rpcz_store.cc:276] 0807 03:39:00.939586 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:01.310468 (+370882us) service_pool.cc:227] Handling call
0807 03:39:01.322607 (+ 12139us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:01.430836 (+108229us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:01.430866 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:01.430915 (+ 49us) raft_consensus.cc:1631] Marking committed up to 1
0807 03:39:01.430942 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:02.413916 (+982974us) spinlock_profiling.cc:243] Waited 108 ms on lock 0x7b54002b43d4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:39:02.413965 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:02.414085 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":248385280}
I20260807 03:39:02.406131 19945 consensus_queue.cc:237] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:02.490360 20275 raft_consensus.cc:482] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:02.486018 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:02.527604 18189 raft_consensus.cc:2454] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 4.
I20260807 03:39:02.536752 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:02.531133 20072 leader_election.cc:288] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
W20260807 03:39:02.537295 20295 log.cc:924] Time spent T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.064s user 0.001s sys 0.000s
I20260807 03:39:02.528951 18027 leader_election.cc:302] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:02.537851 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:02.481642 20272 raft_consensus.cc:2726] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:02.512631 20075 raft_consensus.cc:482] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:02.539037 20272 raft_consensus.cc:2726] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:02.539366 20075 raft_consensus.cc:504] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:02.540191 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:02.540448 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:02.541352 20075 leader_election.cc:288] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:02.541831 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:02.542081 18027 leader_election.cc:302] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:02.543150 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:02.543237 18027 leader_election.cc:302] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:02.543612 20068 raft_consensus.cc:2726] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:02.544237 20068 raft_consensus.cc:2781] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:39:02.544494 18027 leader_election.cc:302] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:02.528620 20042 raft_consensus.cc:2726] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:02.545516 20291 raft_consensus.cc:2726] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:02.545717 18027 leader_election.cc:302] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:02.544632 20068 raft_consensus.cc:482] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:02.546388 20068 raft_consensus.cc:3037] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:02.547325 18027 leader_election.cc:302] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:02.548070 20144 raft_consensus.cc:2726] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:02.541271 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:02.550284 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:02.552810 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:02.553665 18027 leader_election.cc:302] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:02.556334 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:02.556684 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:02.557647 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:02.523906 20133 raft_consensus.cc:482] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:02.529237 20169 raft_consensus.cc:3037] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:02.560890 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:02.561479 18305 raft_consensus.cc:2454] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3.
I20260807 03:39:02.548674 20144 raft_consensus.cc:2726] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:02.529469 20034 leader_election.cc:288] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:02.529465 20254 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:02.563695 18027 leader_election.cc:302] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:02.489624 20286 consensus_queue.cc:1035] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.069s
I20260807 03:39:02.567708 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:02.556947 18190 raft_consensus.cc:2454] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:39:02.535733 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:02.551882 20173 leader_election.cc:288] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:02.554751 19295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:02.558283 18184 raft_consensus.cc:2362] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3. Re-sending same reply.
I20260807 03:39:02.571220 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:02.549321 20275 raft_consensus.cc:504] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:02.574496 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:02.567816 20042 raft_consensus.cc:2726] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:02.600394 18311 raft_consensus.cc:2454] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3.
I20260807 03:39:02.601402 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:02.576226 20073 leader_election.cc:288] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:02.585727 18079 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33268 (request call id 854) took 1003 ms. Trace:
I20260807 03:39:02.603279 18079 rpcz_store.cc:276] 0807 03:39:01.582778 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:01.582963 (+ 185us) service_pool.cc:227] Handling call
0807 03:39:01.993698 (+410735us) spinlock_profiling.cc:243] Waited 9.81 ms on lock 0x7b5800001298. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f25f851 00007fe98f25f702 00007fe99814fb09 00007fe99708b730 00007fe99715179b 00007fe997151821 00007fe9971152b0 00007fe9971007f9 00007fe9971002a2 00007fe9907e6684
0807 03:39:01.993904 (+ 206us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:01.994286 (+ 382us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:01.994322 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:39:02.109889 (+115567us) log.cc:842] Serialized 26 byte log entry
0807 03:39:02.199313 (+ 89424us) raft_consensus.cc:1631] Marking committed up to 1
0807 03:39:02.199350 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:02.199416 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:02.579336 (+379920us) raft_consensus.cc:1671] finished
0807 03:39:02.579378 (+ 42us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:02.585640 (+ 6262us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":18336,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":8674,"spinlock_wait_cycles":22567296,"thread_start_us":30734,"threads_started":1,"wal-append.queue_time_us":156191}
I20260807 03:39:02.569604 18027 leader_election.cc:302] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:02.533897 20250 raft_consensus.cc:504] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:02.592046 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:02.589359 20273 log.cc:924] Time spent T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.786s user 0.004s sys 0.000s
I20260807 03:39:02.579107 20144 raft_consensus.cc:2781] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:02.621610 20144 raft_consensus.cc:482] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:02.622129 20144 raft_consensus.cc:3037] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:02.591663 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:02.629458 18183 raft_consensus.cc:2454] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3.
I20260807 03:39:02.565081 20133 raft_consensus.cc:504] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:02.617506 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:02.617444 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:02.646370 18311 raft_consensus.cc:2454] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:39:02.586117 20247 leader_election.cc:288] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:02.635215 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:02.571229 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:02.657440 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:02.608825 19295 raft_consensus.cc:2454] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
I20260807 03:39:02.662469 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:02.663143 18184 raft_consensus.cc:2454] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:39:02.663501 18309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57268 (request call id 879) took 1726 ms. Trace:
I20260807 03:39:02.666759 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:02.670600 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:02.671250 18183 raft_consensus.cc:2379] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 5: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:02.671867 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:02.663831 18309 rpcz_store.cc:276] 0807 03:39:00.937305 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:01.307790 (+370485us) service_pool.cc:227] Handling call
0807 03:39:02.024877 (+717087us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:02.025036 (+ 159us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:02.025059 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:02.025095 (+ 36us) raft_consensus.cc:1631] Marking committed up to 3
0807 03:39:02.025121 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:02.025167 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:02.663347 (+638180us) spinlock_profiling.cc:243] Waited 717 ms on lock 0x7b540035a050. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bea09 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:02.663447 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1648466816}
I20260807 03:39:02.669317 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:02.676596 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:02.663964 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:02.677695 18190 raft_consensus.cc:2396] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260807 03:39:02.682271 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 112740us
I20260807 03:39:02.602751 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:02.684887 20299 consensus_queue.cc:1046] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260807 03:39:02.685400 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 133903us
I20260807 03:39:02.686584 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:02.686705 18249 leader_election.cc:302] T af8877c3fbba4ab18765832f1390921f P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:39:02.687564 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:02.690104 18189 raft_consensus.cc:2454] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
I20260807 03:39:02.690661 18027 leader_election.cc:302] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:02.694741 18311 raft_consensus.cc:2379] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 4: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:02.695112 18189 raft_consensus.cc:1261] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:39:02.695222 18027 leader_election.cc:302] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:02.695883 18249 leader_election.cc:302] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:02.696734 18027 leader_election.cc:302] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:02.697453 20207 leader_election.cc:288] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:02.698041 18185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44662 (request call id 560) took 3067 ms. Trace:
I20260807 03:39:02.698642 20161 raft_consensus.cc:2726] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:02.698570 18185 rpcz_store.cc:276] 0807 03:38:59.627523 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:59.627659 (+ 136us) service_pool.cc:227] Handling call
0807 03:38:59.779577 (+151918us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:38:59.780429 (+ 852us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:02.694613 (+2914184us) spinlock_profiling.cc:243] Waited 151 ms on lock 0x7b54003c9e50. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bea09 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:02.694740 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":348440320}
I20260807 03:39:02.699121 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:02.699715 18193 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44662 (request call id 574) took 1290 ms. Trace:
I20260807 03:39:02.689934 20042 raft_consensus.cc:2726] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:02.700515 20150 raft_consensus.cc:2781] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:02.700904 20150 raft_consensus.cc:482] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:02.701210 20150 raft_consensus.cc:3037] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:02.698017 20291 raft_consensus.cc:2781] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:02.700290 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:02.691716 20173 raft_consensus.cc:482] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:02.725288 18311 raft_consensus.cc:3037] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:02.700129 18193 rpcz_store.cc:276] 0807 03:39:01.409239 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:01.462379 (+ 53140us) service_pool.cc:227] Handling call
0807 03:39:01.737992 (+275613us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:01.926506 (+188514us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:02.699529 (+773023us) spinlock_profiling.cc:243] Waited 221 ms on lock 0x7b5400172d50. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bea09 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:02.699662 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":507953024}
W20260807 03:39:02.727115 18298 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57276 (request call id 588) took 2392 ms (client timeout 1588 ms). Trace:
W20260807 03:39:02.727506 18298 rpcz_store.cc:269] 0807 03:39:00.334381 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:00.565121 (+230740us) service_pool.cc:227] Handling call
0807 03:39:02.726917 (+2161796us) spinlock_profiling.cc:243] Waited 677 ms on lock 0x7b54001c8754. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:02.727058 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1557658880}
I20260807 03:39:02.690321 18310 raft_consensus.cc:2454] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3.
I20260807 03:39:02.694772 20119 raft_consensus.cc:2726] T af8877c3fbba4ab18765832f1390921f P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:02.735075 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:02.743189 18193 raft_consensus.cc:1261] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:39:02.744908 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:02.724625 20042 raft_consensus.cc:2781] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:39:02.719357 20291 raft_consensus.cc:482] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:02.726799 20207 raft_consensus.cc:482] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:02.729377 20161 raft_consensus.cc:482] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:02.760800 18027 leader_election.cc:302] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:02.762607 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:02.756979 20136 consensus_queue.cc:1035] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260807 03:39:02.740612 20034 raft_consensus.cc:482] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:02.765522 19917 raft_consensus.cc:504] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:02.732759 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:02.715934 20173 raft_consensus.cc:504] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:02.771461 18311 raft_consensus.cc:2454] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3.
I20260807 03:39:02.774148 18300 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57276 (request call id 598) took 1912 ms. Trace:
I20260807 03:39:02.775517 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:02.777486 18300 rpcz_store.cc:276] 0807 03:39:00.861940 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:00.905619 (+ 43679us) service_pool.cc:227] Handling call
0807 03:39:00.905937 (+ 318us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:00.923021 (+ 17084us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:02.773830 (+1850809us) spinlock_profiling.cc:243] Waited 13.7 ms on lock 0x7b5400302fd4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:39:02.774073 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":31560576}
I20260807 03:39:02.759706 20291 raft_consensus.cc:3037] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:02.751817 20042 raft_consensus.cc:482] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:02.778198 18027 leader_election.cc:302] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:02.781801 18311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57268 (request call id 885) took 1331 ms. Trace:
I20260807 03:39:02.770785 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:02.785163 18185 raft_consensus.cc:1226] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 1. Current term is 2. Ops: []
I20260807 03:39:02.786231 20042 raft_consensus.cc:3037] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:02.771828 20272 raft_consensus.cc:2781] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:02.784210 18311 rpcz_store.cc:276] 0807 03:39:01.450182 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:02.780745 (+1330563us) service_pool.cc:227] Handling call
0807 03:39:02.781174 (+ 429us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:02.781408 (+ 234us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:02.781443 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:02.781511 (+ 68us) raft_consensus.cc:1631] Marking committed up to 2
0807 03:39:02.781551 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:02.781613 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:02.781735 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 03:39:02.802613 18298 raft_consensus.cc:2454] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 5.
I20260807 03:39:02.809533 17920 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35098 (request call id 22) took 4179 ms. Trace:
I20260807 03:39:02.810338 18298 raft_consensus.cc:1226] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 2 LEADER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 1. Current term is 2. Ops: []
I20260807 03:39:02.811707 18305 raft_consensus.cc:1226] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:02.812539 18305 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57276 (request call id 614) took 1107 ms. Trace:
I20260807 03:39:02.812918 18305 rpcz_store.cc:276] 0807 03:39:01.705614 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:02.781233 (+1075619us) service_pool.cc:227] Handling call
0807 03:39:02.811491 (+ 30258us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:02.812254 (+ 763us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:02.812480 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 03:39:02.779268 20133 leader_election.cc:288] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:02.809952 17920 rpcz_store.cc:276] 0807 03:38:58.630432 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:58.630657 (+ 225us) service_pool.cc:227] Handling call
0807 03:39:02.809473 (+4178816us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 03:38:58.676930 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 03:38:58.677570 (+ 640us) write_op.cc:432] Acquiring schema lock in shared mode
0807 03:38:58.677602 (+ 32us) write_op.cc:435] Acquired schema lock
0807 03:38:58.677626 (+ 24us) tablet.cc:660] Decoding operations
0807 03:38:58.769399 (+ 91773us) write_op.cc:620] Acquiring the partition lock for write op
0807 03:38:58.771098 (+ 1699us) write_op.cc:641] Partition lock acquired for write op
0807 03:38:58.774383 (+ 3285us) tablet.cc:683] Acquiring locks for 2 operations
0807 03:38:58.801214 (+ 26831us) tablet.cc:699] Row locks acquired
0807 03:38:58.801238 (+ 24us) write_op.cc:260] PREPARE: finished
0807 03:38:58.801409 (+ 171us) write_op.cc:270] Start()
0807 03:38:58.801524 (+ 115us) write_op.cc:276] Timestamp: P: 1786073938801368 usec, L: 0
0807 03:38:58.801543 (+ 19us) op_driver.cc:331] REPLICATION: starting
0807 03:38:58.802186 (+ 643us) log.cc:842] Serialized 988 byte log entry
0807 03:38:59.189069 (+386883us) op_driver.cc:444] REPLICATION: finished
0807 03:38:59.344845 (+155776us) write_op.cc:301] APPLY: starting
0807 03:38:59.344947 (+ 102us) tablet.cc:1365] starting BulkCheckPresence
0807 03:38:59.345095 (+ 148us) tablet.cc:1368] finished BulkCheckPresence
0807 03:38:59.345266 (+ 171us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 03:38:59.470000 (+124734us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 03:38:59.510193 (+ 40193us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0807 03:38:59.523397 (+ 13204us) write_op.cc:312] APPLY: finished
0807 03:38:59.549306 (+ 25909us) log.cc:842] Serialized 28 byte log entry
0807 03:38:59.567688 (+ 18382us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:38:59.573118 (+ 5430us) write_op.cc:454] Released schema lock
0807 03:38:59.575926 (+ 2808us) write_op.cc:341] FINISH: Updating metrics
0807 03:38:59.637917 (+ 61991us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:38:59.646165 (+ 8248us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1563,"reported_tablets":2,"child_traces":[["op",{"apply.queue_time_us":135636,"apply.run_cpu_time_us":13707,"apply.run_wall_time_us":319097,"num_ops":2,"prepare.queue_time_us":35771,"prepare.run_cpu_time_us":3166,"prepare.run_wall_time_us":125520,"replication_time_us":340842,"thread_start_us":70786,"threads_started":1}]]}
I20260807 03:39:02.804136 20214 raft_consensus.cc:504] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:02.827437 18284 tablet_service.cc:1514] Processing DeleteTablet for tablet af8877c3fbba4ab18765832f1390921f with delete_type TABLET_DATA_TOMBSTONED (TS 66a36e1ac8e945e893668e282d394c49 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57260
I20260807 03:39:02.823829 20214 leader_election.cc:288] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:02.774108 19240 leader_election.cc:302] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:02.773634 20275 leader_election.cc:288] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:02.768682 20207 raft_consensus.cc:504] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:02.782584 20219 raft_consensus.cc:482] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:02.776441 20034 raft_consensus.cc:504] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:02.784699 20062 leader_election.cc:288] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:02.852959 18298 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57276 (request call id 616) took 1142 ms. Trace:
I20260807 03:39:02.787420 20159 raft_consensus.cc:2781] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:39:02.771903 20161 raft_consensus.cc:504] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:02.861825 18298 rpcz_store.cc:276] 0807 03:39:01.710215 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:02.809642 (+1099427us) service_pool.cc:227] Handling call
0807 03:39:02.810102 (+ 460us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:02.852677 (+ 42575us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:02.852895 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 03:39:02.828755 18293 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57276 (request call id 599) took 1966 ms. Trace:
I20260807 03:39:02.862530 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:02.843587 18248 leader_election.cc:302] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:02.869474 18293 rpcz_store.cc:276] 0807 03:39:00.862236 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:00.906503 (+ 44267us) service_pool.cc:227] Handling call
0807 03:39:00.906795 (+ 292us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:00.937463 (+ 30668us) raft_consensus.cc:1520] Early marking committed up to index 2
0807 03:39:01.178900 (+241437us) log.cc:842] Serialized 14 byte log entry
0807 03:39:01.366914 (+188014us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:01.366979 (+ 65us) raft_consensus.cc:1631] Marking committed up to 2
0807 03:39:01.367018 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:02.828522 (+1461504us) spinlock_profiling.cc:243] Waited 30.4 ms on lock 0x7b54000d0f54. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:39:02.828572 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:02.828693 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":322671,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":86,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":4752,"spinlock_wait_cycles":70028160,"thread_start_us":418231,"threads_started":2,"wal-append.queue_time_us":271119,"wal-append.run_cpu_time_us":2614,"wal-append.run_wall_time_us":1214786}
W20260807 03:39:02.838943 20281 consensus_meta.cc:287] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.623s user 0.009s sys 0.005s
I20260807 03:39:02.790517 20250 leader_election.cc:288] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:02.804153 20272 raft_consensus.cc:482] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 03:39:02.882990 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 99274us
I20260807 03:39:02.885216 18293 raft_consensus.cc:1226] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20260807 03:39:02.831776 19924 consensus_meta.cc:287] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.592s user 0.007s sys 0.020s
I20260807 03:39:02.837630 19917 leader_election.cc:288] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:02.827840 20204 raft_consensus.cc:504] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:02.861204 20111 raft_consensus.cc:482] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:02.829682 20154 raft_consensus.cc:482] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:02.894454 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:02.895486 18027 leader_election.cc:302] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
W20260807 03:39:02.896183 19240 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 139142us
I20260807 03:39:02.900993 18298 raft_consensus.cc:1226] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:02.899782 18248 leader_election.cc:302] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:02.889420 20048 raft_consensus.cc:2781] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:39:02.886891 20222 raft_consensus.cc:2726] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 03:39:02.865043 19832 consensus_queue.cc:1035] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20260807 03:39:02.879982 20159 raft_consensus.cc:686] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:02.891036 20281 raft_consensus.cc:2726] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:02.861771 20219 raft_consensus.cc:504] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:02.894042 20272 raft_consensus.cc:3037] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:02.924798 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:02.899223 19743 raft_consensus.cc:482] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:02.928646 18069 raft_consensus.cc:2454] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:39:02.930018 18180 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44662 (request call id 567) took 1578 ms. Trace:
I20260807 03:39:02.909988 19496 raft_consensus.cc:2781] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:02.933409 18180 rpcz_store.cc:276] 0807 03:39:01.351734 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:01.351889 (+ 155us) service_pool.cc:227] Handling call
0807 03:39:02.929139 (+1577250us) spinlock_profiling.cc:243] Waited 4.73 ms on lock 0x7b5800003998. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f25f851 00007fe98f25f702 00007fe99814fb09 00007fe99708b730 00007fe99715179b 00007fe997151821 00007fe9971152b0 00007fe9971007f9 00007fe9971002a2 00007fe9907e6684
0807 03:39:02.929402 (+ 263us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:02.929596 (+ 194us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:02.929629 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:02.929679 (+ 50us) raft_consensus.cc:1631] Marking committed up to 2
0807 03:39:02.929717 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:02.929806 (+ 89us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:02.929927 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10874752}
I20260807 03:39:02.917531 20250 raft_consensus.cc:2781] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election won for term 5
I20260807 03:39:02.913810 20048 raft_consensus.cc:482] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:02.895910 19924 raft_consensus.cc:504] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:02.898150 20243 raft_consensus.cc:504] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:02.909843 20111 raft_consensus.cc:504] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:02.937572 19743 raft_consensus.cc:3037] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:02.967878 18180 raft_consensus.cc:1226] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 1. Current term is 2. Ops: []
W20260807 03:39:02.946382 20177 consensus_meta.cc:287] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 0.546s user 0.007s sys 0.008s
I20260807 03:39:02.938905 19496 raft_consensus.cc:482] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:02.972807 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:02.930802 20154 raft_consensus.cc:504] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:02.975536 20281 consensus_queue.cc:1046] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:39:02.976321 20173 leader_election.cc:288] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:02.976658 20281 raft_consensus.cc:3032] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:39:02.977048 20281 raft_consensus.cc:729] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:02.977929 20281 consensus_queue.cc:260] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:02.934351 19471 consensus_queue.cc:1035] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.036s
I20260807 03:39:02.979069 20281 raft_consensus.cc:3037] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
W20260807 03:39:02.983582 19874 consensus_meta.cc:287] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 0.578s user 0.001s sys 0.010s
I20260807 03:39:02.984506 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:02.985469 19924 leader_election.cc:288] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:02.987262 20243 leader_election.cc:288] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
W20260807 03:39:02.988219 20205 consensus_meta.cc:287] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.790s user 0.011s sys 0.000s
I20260807 03:39:02.988629 20205 raft_consensus.cc:504] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:02.988991 20169 raft_consensus.cc:504] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:02.990455 20161 leader_election.cc:288] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:02.990752 20169 leader_election.cc:288] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:02.991030 18309 raft_consensus.cc:2379] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 4: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:02.991472 20205 leader_election.cc:288] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:02.943281 20252 consensus_queue.cc:1046] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.049s
I20260807 03:39:02.950541 20048 raft_consensus.cc:3037] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:02.996155 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
W20260807 03:39:03.024834 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 92094.4us
I20260807 03:39:03.022112 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:03.026093 18180 raft_consensus.cc:3037] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:02.964232 20056 raft_consensus.cc:482] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:03.034955 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:02.992705 19496 raft_consensus.cc:3037] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:02.972312 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:03.043246 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:03.046096 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:03.049304 18188 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44662 (request call id 566) took 1698 ms. Trace:
I20260807 03:39:03.051370 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:03.049619 18188 rpcz_store.cc:276] 0807 03:39:01.350671 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:01.350777 (+ 106us) service_pool.cc:227] Handling call
0807 03:39:01.351097 (+ 320us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:01.368123 (+ 17026us) raft_consensus.cc:1520] Early marking committed up to index 1
0807 03:39:01.368151 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:39:01.406590 (+ 38439us) log.cc:842] Serialized 26 byte log entry
0807 03:39:03.041666 (+1635076us) spinlock_profiling.cc:243] Waited 1.01 ms on lock 0x7b1800249a60. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe9981368b1 00007fe998143661 00007fe991e4e82c 00007fe98f2799dc 00007fe9980b0a39 00007fe98f2968bb 00007fe98f294cbf 00007fe98f296e22 00007fe991e405bb 00007fe991e46063 00007fe991e46411
0807 03:39:03.041837 (+ 171us) raft_consensus.cc:1631] Marking committed up to 1
0807 03:39:03.041884 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:03.041981 (+ 97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:03.042038 (+ 57us) raft_consensus.cc:1671] finished
0807 03:39:03.042051 (+ 13us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:03.049250 (+ 7199us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23435,"raft.queue_time_us":105171,"raft.run_cpu_time_us":5871,"raft.run_wall_time_us":44677,"spinlock_wait_cycles":2317568,"thread_start_us":109082,"threads_started":1,"wal-append.queue_time_us":331308,"wal-append.run_cpu_time_us":2901,"wal-append.run_wall_time_us":1191910}
I20260807 03:39:02.983984 19874 raft_consensus.cc:504] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:02.992292 20177 raft_consensus.cc:504] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:02.983582 20135 raft_consensus.cc:504] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:03.042825 20124 raft_consensus.cc:2726] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:03.065061 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:03.043018 20252 raft_consensus.cc:482] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:03.058383 18072 raft_consensus.cc:2454] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
I20260807 03:39:03.017313 20150 raft_consensus.cc:504] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:03.076195 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:03.082722 18188 raft_consensus.cc:3037] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:03.031586 20056 raft_consensus.cc:504] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:03.086859 20056 leader_election.cc:288] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:03.087196 20150 leader_election.cc:288] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:03.076261 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:03.058573 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:03.106115 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:03.019915 20159 consensus_queue.cc:237] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:03.063469 20207 leader_election.cc:288] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:03.088039 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:03.093464 19874 leader_election.cc:288] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:03.095834 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:03.118355 18189 raft_consensus.cc:3037] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:03.055694 20156 raft_consensus.cc:482] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:03.100020 18180 raft_consensus.cc:2454] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
I20260807 03:39:03.115299 18082 raft_consensus.cc:2454] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
I20260807 03:39:03.121104 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:03.078516 20204 leader_election.cc:288] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:03.117336 19297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331"
I20260807 03:39:03.079279 20111 leader_election.cc:288] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:03.081559 20154 leader_election.cc:288] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:03.124926 18249 leader_election.cc:302] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:03.137617 18189 raft_consensus.cc:2454] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:03.138795 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:03.081074 20252 raft_consensus.cc:504] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:03.141367 18082 raft_consensus.cc:2454] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
I20260807 03:39:03.147074 18188 raft_consensus.cc:2454] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
I20260807 03:39:03.121017 20219 leader_election.cc:288] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:03.151221 18249 leader_election.cc:302] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:03.155524 18080 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33254 (request call id 929) took 1551 ms. Trace:
I20260807 03:39:03.138320 19297 raft_consensus.cc:3037] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:03.155890 18080 rpcz_store.cc:276] 0807 03:39:01.604699 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:01.664767 (+ 60068us) service_pool.cc:227] Handling call
0807 03:39:03.155366 (+1490599us) spinlock_profiling.cc:243] Waited 160 ms on lock 0x7b5400340654. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:03.155469 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":368834304}
I20260807 03:39:03.155913 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:03.160084 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:03.160730 18079 raft_consensus.cc:2454] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
W20260807 03:39:03.148458 20167 consensus_meta.cc:287] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 0.754s user 0.009s sys 0.002s
I20260807 03:39:03.143054 20311 tablet_replica.cc:330] T af8877c3fbba4ab18765832f1390921f P 66a36e1ac8e945e893668e282d394c49: stopping tablet replica
I20260807 03:39:03.167001 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:03.169476 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:03.170185 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:03.170522 18075 raft_consensus.cc:2396] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 03:39:03.171324 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:03.171486 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:03.171864 18069 raft_consensus.cc:3037] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:03.174476 18305 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57276 (request call id 620) took 1412 ms. Trace:
I20260807 03:39:03.175983 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:03.123086 20156 raft_consensus.cc:504] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:03.154703 19659 raft_consensus.cc:2781] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 03:39:03.174870 18305 rpcz_store.cc:276] 0807 03:39:01.761650 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:02.813782 (+1052132us) service_pool.cc:227] Handling call
0807 03:39:02.865043 (+ 51261us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:02.865249 (+ 206us) raft_consensus.cc:1520] Early marking committed up to index 2
0807 03:39:03.065966 (+200717us) log.cc:842] Serialized 14 byte log entry
0807 03:39:03.174122 (+108156us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:03.174176 (+ 54us) raft_consensus.cc:1631] Marking committed up to 2
0807 03:39:03.174220 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:03.174300 (+ 80us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:03.174409 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14992,"thread_start_us":274224,"threads_started":2}
I20260807 03:39:03.181732 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
W20260807 03:39:03.157738 20206 consensus_meta.cc:287] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 1.032s user 0.012s sys 0.005s
I20260807 03:39:03.184285 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:03.184437 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:03.158610 20299 raft_consensus.cc:2781] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 03:39:03.186357 18081 raft_consensus.cc:3037] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:03.207554 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:03.210218 18073 raft_consensus.cc:3037] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:03.213562 18069 raft_consensus.cc:2454] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
I20260807 03:39:03.214824 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:03.218122 20156 leader_election.cc:288] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:03.218725 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:03.219975 18071 raft_consensus.cc:2396] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 03:39:03.220399 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:03.204872 20206 raft_consensus.cc:504] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:03.222433 20206 leader_election.cc:288] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:03.222641 18071 raft_consensus.cc:1226] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:03.223941 19496 raft_consensus.cc:504] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:03.226130 19496 leader_election.cc:288] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:03.226363 20135 leader_election.cc:288] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:03.177021 20167 raft_consensus.cc:504] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:03.228030 19617 raft_consensus.cc:3032] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:39:03.228415 19617 raft_consensus.cc:729] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:03.229997 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:03.230499 18082 raft_consensus.cc:1799] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Handling vote request from an unknown peer 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:03.235375 18082 raft_consensus.cc:2379] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 2: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:03.229184 20167 leader_election.cc:288] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:03.203159 19295 raft_consensus.cc:1226] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [term 2 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 1. Current term is 2. Ops: []
I20260807 03:39:03.243378 18080 raft_consensus.cc:1226] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 4. Current term is 5. Ops: []
I20260807 03:39:03.229185 19617 consensus_queue.cc:260] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:03.238752 20299 raft_consensus.cc:686] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:03.257248 19617 raft_consensus.cc:3037] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:03.251741 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:03.264909 18073 raft_consensus.cc:2454] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
I20260807 03:39:03.244432 20177 leader_election.cc:288] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:03.275442 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:03.281647 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:03.284684 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:03.286357 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:03.286938 18188 raft_consensus.cc:2454] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3.
I20260807 03:39:03.287431 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:03.289677 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:03.290933 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:03.290480 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:03.292639 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:03.292812 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:03.293385 18188 raft_consensus.cc:2379] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 4: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:03.293860 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:39:03.291663 18027 leader_election.cc:334] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057): Timed out: RequestConsensusVote RPC to 127.15.198.130:37057 timed out after 1.693s (SENT)
W20260807 03:39:03.278769 20068 consensus_meta.cc:287] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 0.728s user 0.019s sys 0.011s
W20260807 03:39:03.281316 20018 consensus_meta.cc:287] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 0.908s user 0.006s sys 0.011s
I20260807 03:39:03.307772 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
W20260807 03:39:03.290570 20144 consensus_meta.cc:287] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 0.665s user 0.005s sys 0.010s
I20260807 03:39:03.298266 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:03.330154 18188 raft_consensus.cc:1226] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:03.333351 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:03.345330 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:03.307121 20177 raft_consensus.cc:482] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:03.360780 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:03.361316 20177 raft_consensus.cc:504] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:03.365841 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:03.368157 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:03.340070 19292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331"
I20260807 03:39:03.295130 20252 leader_election.cc:288] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:03.363808 20177 leader_election.cc:288] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:03.401685 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:03.402082 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:03.402948 18293 raft_consensus.cc:1226] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:03.406975 19743 raft_consensus.cc:504] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:03.408452 18298 raft_consensus.cc:2379] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 3: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:03.304621 20068 raft_consensus.cc:504] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:03.415258 20042 consensus_meta.cc:287] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 0.600s user 0.007s sys 0.000s
I20260807 03:39:03.359531 20144 raft_consensus.cc:504] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:03.418314 20144 leader_election.cc:288] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:03.420429 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:03.420802 18079 raft_consensus.cc:2379] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 4: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:03.421212 18311 raft_consensus.cc:2454] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3.
I20260807 03:39:03.303197 20152 raft_consensus.cc:482] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:03.273842 18069 raft_consensus.cc:2454] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:39:03.271418 20272 raft_consensus.cc:504] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:03.391487 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:03.442030 18027 leader_election.cc:302] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
W20260807 03:39:03.442591 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 1.49062e+06us
I20260807 03:39:03.397604 20048 raft_consensus.cc:504] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:03.339205 20062 raft_consensus.cc:482] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:03.357913 20299 consensus_queue.cc:237] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:03.425549 18081 raft_consensus.cc:2396] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260807 03:39:03.427673 20157 consensus_meta.cc:287] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.938s user 0.014s sys 0.002s
I20260807 03:39:03.425276 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:03.429333 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:03.433240 20321 log.cc:924] Time spent T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.144s user 0.000s sys 0.000s
I20260807 03:39:03.433357 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:03.433919 20068 leader_election.cc:288] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:03.437007 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:03.438319 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
W20260807 03:39:03.382398 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 176656us
I20260807 03:39:03.442111 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:03.474097 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:03.395618 20291 consensus_meta.cc:287] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 0.607s user 0.000s sys 0.009s
I20260807 03:39:03.446632 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:03.450522 18073 raft_consensus.cc:2379] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 2: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:03.443491 18027 leader_election.cc:302] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:03.469415 18311 raft_consensus.cc:3037] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:03.472402 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:03.476066 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:03.476678 18073 raft_consensus.cc:3037] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:03.480005 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:03.480377 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:03.480605 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:03.481026 18081 raft_consensus.cc:2454] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:39:03.481819 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:03.483129 18141 leader_election.cc:302] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:03.425808 19297 raft_consensus.cc:2454] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:03.441542 20152 raft_consensus.cc:3037] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:03.415748 20042 raft_consensus.cc:504] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:03.487599 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:03.489449 18073 raft_consensus.cc:2454] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:39:03.498947 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:03.499591 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:03.493696 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:03.500533 18080 raft_consensus.cc:2454] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:39:03.512466 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:03.479669 19583 raft_consensus.cc:482] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:03.469774 20144 raft_consensus.cc:2781] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:03.501698 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:03.500699 20161 raft_consensus.cc:2726] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:03.507254 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:03.449760 20177 consensus_queue.cc:1046] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.062s
W20260807 03:39:03.501303 20318 log.cc:924] Time spent T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.324s user 0.001s sys 0.000s
I20260807 03:39:03.513819 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:03.550985 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:03.551261 20042 leader_election.cc:288] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:03.522394 19583 raft_consensus.cc:504] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:03.515275 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
W20260807 03:39:03.491709 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 48298.2us
I20260807 03:39:03.523725 18069 raft_consensus.cc:2379] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 5: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:03.554174 19583 leader_election.cc:288] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:03.554242 18027 leader_election.cc:302] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:03.554740 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:03.555326 18069 raft_consensus.cc:3037] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:03.509958 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:03.525707 18298 raft_consensus.cc:2454] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:39:03.557356 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:03.557832 20048 leader_election.cc:288] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:03.558331 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:03.559283 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:03.561512 20272 leader_election.cc:288] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:03.565070 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:03.566131 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:03.569514 19743 leader_election.cc:288] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:03.468868 20157 raft_consensus.cc:504] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:03.449533 20062 raft_consensus.cc:504] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:03.588524 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:39:03.622442 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 68302.5us
I20260807 03:39:03.623476 18027 leader_election.cc:302] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:03.625495 18027 leader_election.cc:302] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:03.520251 20144 raft_consensus.cc:482] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:03.573351 19874 raft_consensus.cc:482] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:03.587179 19480 raft_consensus.cc:2726] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:03.641188 18312 raft_consensus.cc:2354] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 4. Current term is 5.
I20260807 03:39:03.609707 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:03.488474 20291 raft_consensus.cc:504] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:03.644953 18080 raft_consensus.cc:2454] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
W20260807 03:39:03.657497 18297 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57276 (request call id 623) took 1794 ms (client timeout 1706 ms). Trace:
I20260807 03:39:03.644320 18298 raft_consensus.cc:2454] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:39:03.665107 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:03.667639 18027 leader_election.cc:302] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
W20260807 03:39:03.663201 18297 rpcz_store.cc:269] 0807 03:39:01.862633 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:02.116627 (+253994us) service_pool.cc:227] Handling call
0807 03:39:03.657328 (+1540701us) spinlock_profiling.cc:243] Waited 50.2 ms on lock 0x7b54001c9154. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:03.657431 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":115399424}
I20260807 03:39:03.669992 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:03.671077 18297 raft_consensus.cc:2396] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260807 03:39:03.541378 18311 raft_consensus.cc:2454] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 4.
I20260807 03:39:03.676570 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:03.677362 18027 leader_election.cc:302] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:03.678035 18311 raft_consensus.cc:1226] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3. Ops: []
I20260807 03:39:03.637504 20042 raft_consensus.cc:2726] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:03.681102 20316 raft_consensus.cc:2726] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 03:39:03.722206 18070 raft_consensus.cc:1226] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:03.724107 18297 raft_consensus.cc:1226] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 1. Current term is 2. Ops: []
I20260807 03:39:03.728113 18069 raft_consensus.cc:2454] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:03.633014 20272 raft_consensus.cc:2781] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:39:03.734035 18297 raft_consensus.cc:1226] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:03.636437 20144 raft_consensus.cc:3037] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
W20260807 03:39:03.750524 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 56761.2us
I20260807 03:39:03.752270 18298 raft_consensus.cc:1226] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:03.755988 18246 leader_election.cc:302] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:03.757639 18027 leader_election.cc:302] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:03.519641 19659 raft_consensus.cc:686] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:03.719323 20291 leader_election.cc:288] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:03.681535 20159 raft_consensus.cc:2781] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:03.766608 20123 raft_consensus.cc:482] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:03.766782 20042 raft_consensus.cc:2726] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:03.681267 20252 raft_consensus.cc:2781] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:39:03.774471 18140 leader_election.cc:302] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:03.750300 20272 raft_consensus.cc:482] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
W20260807 03:39:03.783572 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 294032us
I20260807 03:39:03.784876 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:03.788688 18187 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44662 (request call id 609) took 1236 ms. Trace:
I20260807 03:39:03.789105 18187 rpcz_store.cc:276] 0807 03:39:02.552115 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:03.774861 (+1222746us) service_pool.cc:227] Handling call
0807 03:39:03.788637 (+ 13776us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 03:39:03.789165 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:03.740634 20167 consensus_queue.cc:1035] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20260807 03:39:03.661621 19874 raft_consensus.cc:504] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:03.760787 18027 leader_election.cc:302] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:03.791919 18027 leader_election.cc:302] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:03.792369 18140 leader_election.cc:302] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:03.793339 18140 leader_election.cc:302] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:03.793228 18027 leader_election.cc:302] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:03.777518 20159 raft_consensus.cc:482] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 03:39:03.796280 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 319915us
W20260807 03:39:03.797762 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 314692us
I20260807 03:39:03.774921 20339 raft_consensus.cc:2781] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 03:39:03.801308 18027 leader_election.cc:302] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:03.802680 18027 leader_election.cc:302] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:03.784582 19583 raft_consensus.cc:2726] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:03.781405 20252 raft_consensus.cc:482] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:03.805667 18305 raft_consensus.cc:1226] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:03.787621 20272 raft_consensus.cc:3037] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:03.768265 20316 consensus_queue.cc:1035] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20260807 03:39:03.814225 18141 leader_election.cc:302] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:03.796557 20042 raft_consensus.cc:2781] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:03.801746 20159 raft_consensus.cc:3037] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:03.781220 20123 raft_consensus.cc:504] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:03.787362 20233 consensus_queue.cc:1046] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20260807 03:39:03.805249 20291 raft_consensus.cc:2726] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:03.798308 20177 raft_consensus.cc:2726] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 03:39:03.799729 18248 leader_election.cc:302] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:03.824321 18072 raft_consensus.cc:1226] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3. Ops: []
I20260807 03:39:03.814281 20252 raft_consensus.cc:3037] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:03.815696 20339 raft_consensus.cc:686] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:03.845209 20157 leader_election.cc:288] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:03.847271 20357 raft_consensus.cc:482] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:03.819725 20042 raft_consensus.cc:482] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
W20260807 03:39:03.856732 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 57109.9us
I20260807 03:39:03.832836 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:03.884310 18141 leader_election.cc:302] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:03.828649 20161 raft_consensus.cc:2726] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260807 03:39:03.848521 20340 log.cc:924] Time spent T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.068s user 0.000s sys 0.001s
I20260807 03:39:03.903393 20123 leader_election.cc:288] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:03.852407 20352 raft_consensus.cc:2726] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:03.850167 20154 raft_consensus.cc:2726] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:03.895020 20042 raft_consensus.cc:3037] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:03.897012 20316 raft_consensus.cc:482] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:03.904515 20272 raft_consensus.cc:504] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:03.904968 19874 leader_election.cc:288] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:03.853340 20111 raft_consensus.cc:2726] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:03.905562 20034 leader_election.cc:288] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:03.828472 20350 consensus_queue.cc:1046] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.085s
I20260807 03:39:03.908905 20339 consensus_queue.cc:237] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
W20260807 03:39:03.940261 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 146963us
I20260807 03:39:03.909446 18190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44660 (request call id 1057) took 1040 ms. Trace:
I20260807 03:39:03.942128 18140 leader_election.cc:302] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:03.942144 18190 rpcz_store.cc:276] 0807 03:39:02.868982 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:02.936980 (+ 67998us) service_pool.cc:227] Handling call
0807 03:39:02.940881 (+ 3901us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:02.977069 (+ 36188us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:02.977122 (+ 53us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:39:02.977698 (+ 576us) log.cc:842] Serialized 26 byte log entry
0807 03:39:03.077973 (+100275us) raft_consensus.cc:1631] Marking committed up to 1
0807 03:39:03.078013 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:03.909019 (+831006us) spinlock_profiling.cc:243] Waited 35.7 ms on lock 0x7b54003c9e54. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:39:03.909074 (+ 55us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:03.909129 (+ 55us) raft_consensus.cc:1671] finished
0807 03:39:03.909144 (+ 15us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:03.909365 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1345,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":156,"spinlock_wait_cycles":82174208,"thread_start_us":99849,"threads_started":1,"wal-append.queue_time_us":152581}
I20260807 03:39:03.896844 18141 leader_election.cc:302] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:03.893965 20356 raft_consensus.cc:2781] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:03.847541 20167 raft_consensus.cc:2726] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:03.900820 20265 raft_consensus.cc:2726] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:03.956338 18191 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44660 (request call id 1014) took 2337 ms. Trace:
I20260807 03:39:03.957342 18191 rpcz_store.cc:276] 0807 03:39:01.619153 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:01.933294 (+314141us) service_pool.cc:227] Handling call
0807 03:39:03.955910 (+2022616us) spinlock_profiling.cc:243] Waited 795 ms on lock 0x7b5400012554. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:03.956060 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1828320128}
I20260807 03:39:03.960330 20352 raft_consensus.cc:2781] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:03.961519 18140 leader_election.cc:302] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
W20260807 03:39:03.963519 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 68608.2us
W20260807 03:39:03.963558 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 160952us
I20260807 03:39:03.964706 18027 leader_election.cc:302] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:03.965170 18079 raft_consensus.cc:1226] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 1. Current term is 2. Ops: []
I20260807 03:39:03.965962 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
W20260807 03:39:03.966311 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 69536.3us
I20260807 03:39:03.967042 18141 leader_election.cc:302] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:03.968757 18140 leader_election.cc:302] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:03.926748 20152 raft_consensus.cc:504] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:03.970604 18140 leader_election.cc:302] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:03.970498 18141 leader_election.cc:302] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:03.948284 20356 raft_consensus.cc:482] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:03.976147 19480 raft_consensus.cc:3037] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:03.976285 20356 raft_consensus.cc:3037] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:03.971446 20350 raft_consensus.cc:2726] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:03.974275 20272 leader_election.cc:288] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:03.981760 20124 consensus_queue.cc:1046] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:39:03.982339 20111 raft_consensus.cc:3032] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 03:39:03.982463 20124 consensus_queue.cc:1225] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer d9716e2bf06442f5b3b0e81087c53c4d log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260807 03:39:03.982753 20111 raft_consensus.cc:729] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:03.983546 20111 consensus_queue.cc:260] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:03.971583 18027 leader_election.cc:302] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:03.972604 18072 raft_consensus.cc:1226] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:03.986384 20350 consensus_queue.cc:1046] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:39:03.987306 18072 raft_consensus.cc:1226] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:03.987413 20350 raft_consensus.cc:3032] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 03:39:03.987766 20350 raft_consensus.cc:729] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:03.963649 20352 raft_consensus.cc:482] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:03.988593 20352 raft_consensus.cc:3037] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:03.988623 18072 raft_consensus.cc:1226] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 1. Current term is 2. Ops: []
I20260807 03:39:03.989563 20363 raft_consensus.cc:3032] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:39:03.989914 20363 raft_consensus.cc:729] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:03.990758 20363 consensus_queue.cc:260] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:03.969498 20167 raft_consensus.cc:2726] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:03.974373 20154 raft_consensus.cc:2781] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:39:03.974257 19874 consensus_queue.cc:1046] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:39:03.968192 20365 raft_consensus.cc:2726] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:03.988533 20350 consensus_queue.cc:260] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:03.993125 19874 raft_consensus.cc:3032] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:39:03.993273 20347 raft_consensus.cc:2781] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:39:03.931728 20316 raft_consensus.cc:504] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:03.993574 19874 raft_consensus.cc:729] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:03.993798 20347 raft_consensus.cc:686] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:03.993860 18027 leader_election.cc:302] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:03.994407 19874 consensus_queue.cc:260] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:03.995118 20316 leader_election.cc:288] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:03.970260 20265 raft_consensus.cc:2781] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:03.996332 20265 raft_consensus.cc:482] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:03.996526 20034 raft_consensus.cc:2726] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 03:39:03.996742 20265 raft_consensus.cc:3037] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:03.997828 18027 leader_election.cc:302] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:03.992023 20363 raft_consensus.cc:3037] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:03.994729 20347 consensus_queue.cc:237] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.001549 19480 raft_consensus.cc:2726] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:04.003839 20152 leader_election.cc:288] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:04.004525 19295 raft_consensus.cc:1226] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 [term 2 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 1. Current term is 2. Ops: []
I20260807 03:39:03.996745 19297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:04.004402 18027 leader_election.cc:302] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:04.005322 20167 raft_consensus.cc:2726] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 03:39:04.004782 19480 consensus_queue.cc:1046] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 03:39:04.006127 20177 raft_consensus.cc:2726] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 03:39:03.900728 20357 raft_consensus.cc:504] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:03.992218 20272 raft_consensus.cc:2726] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:04.007614 20124 raft_consensus.cc:3032] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:39:04.008464 18079 raft_consensus.cc:1226] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3. Ops: []
W20260807 03:39:04.008651 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 111895us
I20260807 03:39:04.008709 18249 leader_election.cc:302] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:04.008405 18027 leader_election.cc:302] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:04.020942 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:04.021127 18248 leader_election.cc:302] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:04.021907 18027 leader_election.cc:302] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:03.848062 19659 consensus_queue.cc:237] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.023451 18248 leader_election.cc:302] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:03.995631 19874 raft_consensus.cc:3037] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:04.027933 18191 raft_consensus.cc:1261] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 03:39:04.023567 18027 leader_election.cc:302] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:39:04.008296 20154 raft_consensus.cc:482] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:04.009706 20124 raft_consensus.cc:729] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:04.022604 20175 raft_consensus.cc:2726] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 03:39:04.022588 20272 raft_consensus.cc:2726] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 03:39:04.037529 18027 leader_election.cc:302] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:04.037385 18248 leader_election.cc:302] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:04.039938 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:03.993533 20350 raft_consensus.cc:3037] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:04.024421 20316 raft_consensus.cc:2726] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:04.044410 19853 raft_consensus.cc:2726] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:04.050391 20173 raft_consensus.cc:3032] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Stepping down as leader of term 5
I20260807 03:39:04.050336 19471 consensus_queue.cc:1035] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 03:39:04.051220 19480 consensus_queue.cc:1046] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:39:04.041162 20167 raft_consensus.cc:2726] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 03:39:04.023154 20144 raft_consensus.cc:504] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.029119 20157 raft_consensus.cc:2726] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:04.058981 20173 raft_consensus.cc:729] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:04.076332 20124 consensus_queue.cc:260] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.077917 20352 raft_consensus.cc:504] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.089500 20159 raft_consensus.cc:504] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.095430 18185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44660 (request call id 1034) took 1657 ms. Trace:
I20260807 03:39:04.105834 20177 raft_consensus.cc:2726] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 03:39:04.058359 20252 raft_consensus.cc:504] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.083222 20042 raft_consensus.cc:504] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.086863 20265 raft_consensus.cc:504] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.056589 20353 raft_consensus.cc:2726] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:04.058514 19239 leader_election.cc:302] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:04.107255 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:04.004856 20365 consensus_queue.cc:1046] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.058s
I20260807 03:39:04.082592 20111 raft_consensus.cc:3037] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
W20260807 03:39:04.087363 20310 log.cc:924] Time spent T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 1.056s user 0.000s sys 0.015s
W20260807 03:39:04.110126 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 55008.5us
I20260807 03:39:04.140573 20365 raft_consensus.cc:3032] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:39:04.140964 20365 raft_consensus.cc:729] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:04.141206 20159 leader_election.cc:288] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:04.141414 19239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 79562.3us
I20260807 03:39:04.111279 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:04.141726 20365 consensus_queue.cc:260] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.142921 18070 raft_consensus.cc:1226] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 1. Current term is 2. Ops: []
I20260807 03:39:04.142956 20365 raft_consensus.cc:3037] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
W20260807 03:39:04.112433 18312 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57268 (request call id 891) took 2141 ms (client timeout 1882 ms). Trace:
W20260807 03:39:04.144258 18312 rpcz_store.cc:269] 0807 03:39:01.970958 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:02.278071 (+307113us) service_pool.cc:227] Handling call
0807 03:39:04.112189 (+1834118us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b540018b0d4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:04.112355 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2767960960}
I20260807 03:39:04.145889 18070 raft_consensus.cc:1226] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 3. Current term is 4. Ops: []
W20260807 03:39:04.146920 18309 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57268 (request call id 886) took 2666 ms (client timeout 2910 ms). Trace:
W20260807 03:39:04.147293 18309 rpcz_store.cc:269] 0807 03:39:01.480340 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:02.686363 (+1206023us) service_pool.cc:227] Handling call
0807 03:39:04.146756 (+1460393us) spinlock_profiling.cc:243] Waited 298 ms on lock 0x7b54003e5b54. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:04.146869 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":685147392}
I20260807 03:39:04.149637 18300 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57276 (request call id 603) took 2550 ms. Trace:
I20260807 03:39:04.149410 20356 raft_consensus.cc:504] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.149856 18070 raft_consensus.cc:1226] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 1. Current term is 2. Ops: []
I20260807 03:39:04.151451 20356 leader_election.cc:288] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
W20260807 03:39:04.151649 18295 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57276 (request call id 608) took 2542 ms (client timeout 2113 ms). Trace:
W20260807 03:39:04.152001 18295 rpcz_store.cc:269] 0807 03:39:01.609424 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:02.171555 (+562131us) service_pool.cc:227] Handling call
0807 03:39:04.151467 (+1979912us) spinlock_profiling.cc:243] Waited 582 ms on lock 0x7b54001d0954. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:04.151588 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1339499392}
I20260807 03:39:04.153019 18071 raft_consensus.cc:1226] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 1. Current term is 2. Ops: []
I20260807 03:39:04.155082 20265 leader_election.cc:288] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:04.156301 20124 raft_consensus.cc:3037] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:04.136051 20357 leader_election.cc:288] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:04.158855 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:04.159664 18075 raft_consensus.cc:2379] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 2: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
W20260807 03:39:04.159855 18138 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.132:41381, user_credentials={real_user=slave}} blocked reactor thread for 75226.4us
I20260807 03:39:04.141273 18248 leader_election.cc:302] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:04.161657 18248 leader_election.cc:302] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
W20260807 03:39:04.114574 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 77068.6us
I20260807 03:39:04.121922 18310 raft_consensus.cc:1226] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 3. Current term is 5. Ops: []
I20260807 03:39:04.165864 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:04.123468 20144 leader_election.cc:288] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:04.124506 18081 raft_consensus.cc:2379] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 4: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:04.125689 20352 leader_election.cc:288] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:04.125303 20173 consensus_queue.cc:260] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.127590 18185 rpcz_store.cc:276] 0807 03:39:02.437981 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:02.699744 (+261763us) service_pool.cc:227] Handling call
0807 03:39:02.700010 (+ 266us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:02.788798 (+ 88788us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:04.095180 (+1306382us) spinlock_profiling.cc:243] Waited 85 ms on lock 0x7b54002bfcd4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:39:04.095357 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":195441536}
I20260807 03:39:04.036196 20154 raft_consensus.cc:3037] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:04.169159 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:04.170912 20062 leader_election.cc:288] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:04.171686 20277 raft_consensus.cc:2781] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:04.172129 20277 raft_consensus.cc:482] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:04.172487 20277 raft_consensus.cc:3037] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:04.175050 19480 raft_consensus.cc:3032] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 03:39:04.175452 19480 raft_consensus.cc:729] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:04.176272 19480 consensus_queue.cc:260] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.177477 19480 raft_consensus.cc:3037] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:04.177718 20154 raft_consensus.cc:504] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.179230 18312 raft_consensus.cc:1261] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:39:04.180285 19583 raft_consensus.cc:3032] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:39:04.180644 19583 raft_consensus.cc:729] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:04.181005 20042 leader_election.cc:288] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:04.181170 20154 leader_election.cc:288] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:04.181433 19583 consensus_queue.cc:260] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.182077 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:04.182825 19583 raft_consensus.cc:3037] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:04.182842 18312 raft_consensus.cc:2454] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 2.
I20260807 03:39:04.185402 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:04.186105 18312 raft_consensus.cc:2454] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 4.
I20260807 03:39:04.188360 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
W20260807 03:39:04.190137 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 92041.2us
I20260807 03:39:04.135015 18080 raft_consensus.cc:1261] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:39:04.191082 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:04.136967 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 79967.2us
I20260807 03:39:04.137647 20367 consensus_queue.cc:1046] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:39:04.138325 20152 raft_consensus.cc:2726] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260807 03:39:04.139462 20317 log.cc:924] Time spent T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 1.015s user 0.008s sys 0.000s
W20260807 03:39:04.110937 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 55317.2us
I20260807 03:39:04.149988 18300 rpcz_store.cc:276] 0807 03:39:01.599713 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:02.781111 (+1181398us) service_pool.cc:227] Handling call
0807 03:39:02.781391 (+ 280us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:02.927613 (+146222us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:02.927642 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:02.927677 (+ 35us) raft_consensus.cc:1631] Marking committed up to 1
0807 03:39:02.927701 (+ 24us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:04.149449 (+1221748us) spinlock_profiling.cc:243] Waited 146 ms on lock 0x7b54000ff854. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:39:04.149489 (+ 40us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:04.149591 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3502,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":6066,"spinlock_wait_cycles":335614208}
I20260807 03:39:04.197261 18248 leader_election.cc:302] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:04.197613 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:04.198673 18248 leader_election.cc:302] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:04.194216 18141 leader_election.cc:302] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
W20260807 03:39:04.194656 20250 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.223s user 0.007s sys 0.008s
I20260807 03:39:04.194564 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:04.195552 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:04.201324 18305 raft_consensus.cc:3037] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:04.200508 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:04.202280 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:04.202615 18312 raft_consensus.cc:3037] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:04.203028 18185 raft_consensus.cc:1226] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 1. Current term is 2. Ops: []
I20260807 03:39:04.215443 18080 raft_consensus.cc:2454] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:39:04.216286 18141 leader_election.cc:302] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
W20260807 03:39:04.212283 20250 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.243s user 0.007s sys 0.008s
I20260807 03:39:04.226150 18069 raft_consensus.cc:1226] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:04.230794 18141 leader_election.cc:302] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
W20260807 03:39:04.220818 20208 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.553s user 0.007s sys 0.025s
I20260807 03:39:04.213622 20156 raft_consensus.cc:2781] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:39:03.364223 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:39:04.238247 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:04.238813 18189 raft_consensus.cc:2454] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 2.
I20260807 03:39:04.239588 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:04.240314 18189 raft_consensus.cc:2379] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 5: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:04.240928 18141 leader_election.cc:302] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:04.241297 18305 raft_consensus.cc:2454] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 4.
I20260807 03:39:04.221411 20180 raft_consensus.cc:2726] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 03:39:04.240443 18312 raft_consensus.cc:2454] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 4.
I20260807 03:39:04.217913 19496 raft_consensus.cc:2781] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:04.224388 20363 raft_consensus.cc:2781] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:39:04.246376 18140 leader_election.cc:302] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:04.246747 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:04.222046 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:04.247614 18248 leader_election.cc:302] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:04.221920 20350 raft_consensus.cc:3032] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 03:39:04.205358 20275 consensus_queue.cc:1046] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20260807 03:39:04.235709 20367 raft_consensus.cc:2726] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:04.196061 19861 consensus_queue.cc:1046] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20260807 03:39:04.254285 20347 consensus_queue.cc:1046] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:39:04.231709 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:04.245723 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:04.245878 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:04.225063 20250 raft_consensus.cc:686] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:04.243772 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
W20260807 03:39:04.240641 20208 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.567s user 0.007s sys 0.026s
I20260807 03:39:04.252425 20123 raft_consensus.cc:2781] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:04.236564 20156 raft_consensus.cc:482] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:04.257150 18248 leader_election.cc:302] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:04.244501 19496 raft_consensus.cc:482] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:04.253003 18072 raft_consensus.cc:3037] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:04.271171 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:04.274755 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:04.250869 20350 raft_consensus.cc:729] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:04.275431 18187 raft_consensus.cc:3037] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:04.275846 18180 raft_consensus.cc:2454] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 2.
I20260807 03:39:04.246356 20154 consensus_queue.cc:1046] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20260807 03:39:04.255966 20363 raft_consensus.cc:686] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:04.261307 20357 raft_consensus.cc:2726] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 03:39:04.281913 18184 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44660 (request call id 1046) took 1712 ms. Trace:
I20260807 03:39:04.291260 20124 raft_consensus.cc:3032] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:39:04.293534 18187 raft_consensus.cc:2454] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 4.
I20260807 03:39:04.268314 20123 raft_consensus.cc:482] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:04.272995 20073 raft_consensus.cc:482] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:04.287124 20252 leader_election.cc:288] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:04.289917 18184 rpcz_store.cc:276] 0807 03:39:02.568636 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:02.744512 (+175876us) service_pool.cc:227] Handling call
0807 03:39:02.766662 (+ 22150us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:02.789496 (+ 22834us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:02.803818 (+ 14322us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:39:02.903383 (+ 99565us) log.cc:842] Serialized 26 byte log entry
0807 03:39:03.054846 (+151463us) raft_consensus.cc:1631] Marking committed up to 2
0807 03:39:03.061943 (+ 7097us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:03.067547 (+ 5604us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:04.138917 (+1071370us) spinlock_profiling.cc:243] Waited 2.87 ms on lock 0x7b44000232cc. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f1a9a95 00007fe98f1aa4b2 00007fe991e52511 00007fe991e4de1a 00007fe991e41bee 00007fe991e40f36 00007fe991e40b58 00007fe991e4a551 00007fe991e4a509 00007fe991e4a499
0807 03:39:04.255376 (+116459us) raft_consensus.cc:1671] finished
0807 03:39:04.260300 (+ 4924us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:04.280606 (+ 20306us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24986,"raft.queue_time_us":2549,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":4692,"spinlock_wait_cycles":6592768,"thread_start_us":102974,"threads_started":1,"wal-append.queue_time_us":162638}
I20260807 03:39:04.274088 20156 raft_consensus.cc:3037] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:04.291657 20124 raft_consensus.cc:729] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:04.273797 20233 consensus_queue.cc:1046] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.057s
I20260807 03:39:04.285204 19945 consensus_queue.cc:1046] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:39:04.298942 18180 raft_consensus.cc:1261] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 03:39:04.274641 19496 raft_consensus.cc:3037] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:04.268615 18183 raft_consensus.cc:2454] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
I20260807 03:39:04.337248 18248 leader_election.cc:302] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:04.331288 20123 raft_consensus.cc:3037] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:04.348968 18188 raft_consensus.cc:1226] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 2. Current term is 3. Ops: []
I20260807 03:39:04.272130 20365 raft_consensus.cc:2726] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:04.273416 20048 consensus_queue.cc:1046] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.059s
I20260807 03:39:04.295430 18187 raft_consensus.cc:1226] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 3. Current term is 4. Ops: [2.2-3.3]
I20260807 03:39:04.358157 18248 leader_election.cc:302] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
W20260807 03:39:04.346297 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 98924.2us
I20260807 03:39:04.330468 19847 raft_consensus.cc:2726] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260807 03:39:04.368698 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 167207us
I20260807 03:39:04.369596 18027 leader_election.cc:302] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:04.370216 18248 leader_election.cc:302] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:04.370760 18027 leader_election.cc:302] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:04.371757 18248 leader_election.cc:302] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:04.371974 18141 leader_election.cc:398] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021) with higher term. Message: Invalid argument: T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 4. Current term is 5.
I20260807 03:39:04.372941 18141 leader_election.cc:302] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:04.376240 20376 raft_consensus.cc:2781] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader election won for term 4
I20260807 03:39:04.376809 20376 raft_consensus.cc:686] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:04.377626 20376 consensus_queue.cc:237] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.380259 20149 raft_consensus.cc:2726] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:04.380874 20180 consensus_queue.cc:1046] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:39:04.381752 20180 raft_consensus.cc:2781] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:39:04.382238 20180 raft_consensus.cc:686] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 2 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:04.383112 20180 consensus_queue.cc:237] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.384271 20181 raft_consensus.cc:3032] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 03:39:04.384799 20181 raft_consensus.cc:729] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:04.374507 20252 raft_consensus.cc:2726] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 03:39:04.385830 20181 consensus_queue.cc:260] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.387414 20181 raft_consensus.cc:3037] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:04.394208 18027 leader_election.cc:302] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:04.368945 20119 raft_consensus.cc:2781] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 03:39:04.349364 20073 raft_consensus.cc:504] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.402745 18027 leader_election.cc:302] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:04.404537 18027 leader_election.cc:302] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:04.405594 20159 raft_consensus.cc:2726] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:04.408634 20119 raft_consensus.cc:686] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:04.408967 20048 raft_consensus.cc:3032] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:39:04.409291 20048 raft_consensus.cc:729] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:04.409551 20119 consensus_queue.cc:237] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.410041 18027 leader_election.cc:302] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:04.411445 20177 raft_consensus.cc:2781] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader election won for term 4
I20260807 03:39:04.411641 20073 leader_election.cc:288] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:04.412067 20177 raft_consensus.cc:686] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
W20260807 03:39:04.413195 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 45845.2us
I20260807 03:39:04.413440 20177 consensus_queue.cc:237] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.415059 18249 leader_election.cc:302] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:04.416306 19480 raft_consensus.cc:2726] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260807 03:39:04.416409 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 44690.3us
I20260807 03:39:04.417339 20299 consensus_queue.cc:1046] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:39:04.375023 20363 consensus_queue.cc:237] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.418362 20299 raft_consensus.cc:2726] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:04.421309 18027 leader_election.cc:302] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:04.424363 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:04.426513 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:04.418519 20149 raft_consensus.cc:3032] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:39:04.429584 18295 raft_consensus.cc:1226] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:04.423668 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:04.425872 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:04.430441 18309 raft_consensus.cc:2454] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:39:04.428035 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:04.398677 20062 consensus_queue.cc:1035] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.041s
I20260807 03:39:04.417727 20204 raft_consensus.cc:2781] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:39:04.440385 20204 raft_consensus.cc:482] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:04.440636 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:04.440861 20204 raft_consensus.cc:3037] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:04.441344 18071 raft_consensus.cc:3037] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:04.444528 20018 raft_consensus.cc:482] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:04.445245 18305 raft_consensus.cc:1226] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:04.445142 20018 raft_consensus.cc:504] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.446557 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:04.447155 20018 leader_election.cc:288] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:04.447332 18305 raft_consensus.cc:2454] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 4.
I20260807 03:39:04.354171 20350 consensus_queue.cc:260] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.441581 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:04.449723 18298 raft_consensus.cc:3037] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:04.417658 20311 raft_consensus.cc:2229] T af8877c3fbba4ab18765832f1390921f P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:39:04.444445 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:04.451742 18311 raft_consensus.cc:2354] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 4. Current term is 5.
I20260807 03:39:04.452250 18027 leader_election.cc:302] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:04.453398 18141 leader_election.cc:302] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:04.355545 20250 consensus_queue.cc:237] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.368414 20277 raft_consensus.cc:504] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.374877 20357 consensus_queue.cc:1035] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.058s
I20260807 03:39:04.460029 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:04.421197 20144 raft_consensus.cc:2781] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:39:04.429370 20149 raft_consensus.cc:729] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:04.448725 20161 raft_consensus.cc:2781] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader election won for term 4
I20260807 03:39:04.463289 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:04.464087 18141 leader_election.cc:302] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:04.453497 20150 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:04.455533 20042 raft_consensus.cc:2781] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260807 03:39:04.496038 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 246990us
I20260807 03:39:04.391696 20124 consensus_queue.cc:260] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.480592 18298 raft_consensus.cc:2454] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 5.
I20260807 03:39:04.498451 18248 leader_election.cc:302] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:04.499567 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:04.484766 20042 raft_consensus.cc:482] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:04.498004 20352 raft_consensus.cc:2726] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:04.474514 20275 raft_consensus.cc:2781] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:39:04.485116 20161 raft_consensus.cc:686] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:04.484951 20150 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.468039 20144 raft_consensus.cc:482] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:04.473661 20350 raft_consensus.cc:3037] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:04.505923 20275 raft_consensus.cc:482] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:04.504686 19874 raft_consensus.cc:482] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:04.525671 20154 raft_consensus.cc:3032] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:39:04.525465 19874 raft_consensus.cc:504] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.526075 20154 raft_consensus.cc:729] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:04.512987 20144 raft_consensus.cc:3037] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:04.526924 20154 consensus_queue.cc:260] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.527386 19874 leader_election.cc:288] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:04.527951 20154 raft_consensus.cc:3037] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:04.524933 20275 raft_consensus.cc:3037] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:04.529485 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:04.529808 18141 leader_election.cc:398] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Vote denied by peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021) with higher term. Message: Invalid argument: T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 4. Current term is 5.
I20260807 03:39:04.552630 18248 leader_election.cc:302] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:04.556154 18141 leader_election.cc:401] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260807 03:39:04.524427 20356 raft_consensus.cc:2726] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 03:39:04.559906 18141 leader_election.cc:302] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:04.506565 19297 raft_consensus.cc:1226] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 [term 2 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 1. Current term is 2. Ops: []
I20260807 03:39:04.503635 20385 consensus_queue.cc:1046] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.025s
I20260807 03:39:04.501475 19295 raft_consensus.cc:1261] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 5 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260807 03:39:04.531126 18185 raft_consensus.cc:1261] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260807 03:39:04.556460 18190 raft_consensus.cc:2454] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:04.410131 20048 consensus_queue.cc:260] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.509545 19583 raft_consensus.cc:3032] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:39:04.502228 20042 raft_consensus.cc:3037] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:04.573014 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:04.567271 18249 leader_election.cc:302] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:04.500768 20397 raft_consensus.cc:2726] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:04.589419 18300 raft_consensus.cc:1226] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:04.589799 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:04.509747 20150 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:04.597289 18027 leader_election.cc:302] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:04.593497 18309 raft_consensus.cc:1226] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:04.552160 20311 raft_consensus.cc:2258] T af8877c3fbba4ab18765832f1390921f P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:39:04.515401 20173 raft_consensus.cc:3037] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 6
W20260807 03:39:04.632078 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 64875.7us
I20260807 03:39:04.566749 20208 raft_consensus.cc:3037] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:04.572958 19743 raft_consensus.cc:2726] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:04.573160 20111 raft_consensus.cc:2781] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election won for term 5
W20260807 03:39:04.643471 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 46268.9us
I20260807 03:39:04.595511 20265 raft_consensus.cc:3032] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:39:04.558902 20387 consensus_queue.cc:1046] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.080s
I20260807 03:39:04.625766 20177 raft_consensus.cc:482] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:04.637233 20367 raft_consensus.cc:3032] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:39:04.596626 18248 leader_election.cc:302] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:04.587285 19583 raft_consensus.cc:729] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:04.559369 20389 consensus_queue.cc:1046] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.081s
I20260807 03:39:04.643015 20219 raft_consensus.cc:2781] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:39:04.645404 19874 raft_consensus.cc:2726] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:04.507157 20161 consensus_queue.cc:237] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.666589 20299 consensus_queue.cc:1035] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:39:04.667207 18071 raft_consensus.cc:2454] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:39:04.674916 18185 raft_consensus.cc:1261] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260807 03:39:04.565568 20149 consensus_queue.cc:260] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:04.678785 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 82261.1us
I20260807 03:39:04.649331 20265 raft_consensus.cc:729] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:04.559316 20365 consensus_queue.cc:1046] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.103s
I20260807 03:39:04.559818 20156 raft_consensus.cc:504] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.657569 20111 raft_consensus.cc:686] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 5 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:04.662369 20367 raft_consensus.cc:729] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:04.699343 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:04.704169 20356 consensus_queue.cc:1035] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:39:04.704277 20156 leader_election.cc:288] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:04.729143 18300 raft_consensus.cc:1226] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:04.687697 20214 raft_consensus.cc:2726] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:04.705523 20387 raft_consensus.cc:3032] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:39:04.706094 19471 consensus_queue.cc:1035] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 03:39:04.672209 20219 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:04.709878 20150 consensus_queue.cc:1046] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 03:39:04.710300 18248 leader_election.cc:302] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:04.711272 19924 consensus_queue.cc:1046] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:39:04.733955 20062 raft_consensus.cc:482] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:04.731807 20018 raft_consensus.cc:482] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:04.712419 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:04.749574 18070 raft_consensus.cc:1226] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 1. Current term is 2. Ops: []
I20260807 03:39:04.713670 18310 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57276 (request call id 625) took 2762 ms. Trace:
I20260807 03:39:04.712001 20204 raft_consensus.cc:504] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.752568 18309 raft_consensus.cc:1226] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 2 LEADER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 1. Current term is 2. Ops: []
I20260807 03:39:04.717175 20149 raft_consensus.cc:3037] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:04.719866 20144 raft_consensus.cc:504] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.722332 20311 ts_tablet_manager.cc:1905] T af8877c3fbba4ab18765832f1390921f P 66a36e1ac8e945e893668e282d394c49: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 03:39:04.662870 20177 raft_consensus.cc:504] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.724351 17920 catalog_manager.cc:5495] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 reported cstate change: config changed from index 3 to 4, term changed from 1 to 2, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to <none>, VOTER 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) evicted. New cstate: current_term: 2 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:04.673002 20252 raft_consensus.cc:482] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:04.736876 17912 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39492 (request call id 145) took 7358 ms. Trace:
I20260807 03:39:04.763060 18300 raft_consensus.cc:1226] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 1. Current term is 2. Ops: []
I20260807 03:39:04.763670 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:04.764612 18069 raft_consensus.cc:2379] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 3: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:04.683413 20376 raft_consensus.cc:482] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:04.687849 19874 raft_consensus.cc:482] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:04.747785 20181 raft_consensus.cc:2726] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:04.749037 18081 raft_consensus.cc:1226] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 1. Current term is 2. Ops: []
I20260807 03:39:04.769505 18312 raft_consensus.cc:2354] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3.
I20260807 03:39:04.770358 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:04.695538 20395 consensus_queue.cc:1046] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
I20260807 03:39:04.774065 19874 raft_consensus.cc:504] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.776063 19874 leader_election.cc:288] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:04.777426 20144 leader_election.cc:288] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:04.697012 20277 leader_election.cc:288] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:04.737983 20387 raft_consensus.cc:729] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:04.715653 18183 raft_consensus.cc:1226] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 1. Current term is 2. Ops: []
I20260807 03:39:04.744212 20219 raft_consensus.cc:3037] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:04.759825 17920 catalog_manager.cc:5495] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 reported cstate change: config changed from index 3 to 4, VOTER 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) evicted. New cstate: current_term: 1 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:04.801570 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:04.765992 18082 raft_consensus.cc:1226] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:04.717192 19659 raft_consensus.cc:3032] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:39:04.736011 19496 raft_consensus.cc:504] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.749635 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:04.771557 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:04.806030 18080 raft_consensus.cc:3037] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:04.806247 18081 raft_consensus.cc:2379] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 3: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:04.772009 18070 raft_consensus.cc:1226] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 1. Current term is 2. Ops: []
I20260807 03:39:04.779491 18075 raft_consensus.cc:1261] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 03:39:04.779614 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:04.745512 20347 consensus_queue.cc:1046] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
I20260807 03:39:04.733192 19583 consensus_queue.cc:260] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.811045 18248 leader_election.cc:302] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:04.796875 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:04.806993 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:04.805020 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:04.808821 18300 raft_consensus.cc:2379] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 2: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:04.767849 20124 raft_consensus.cc:3037] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:04.762689 17912 rpcz_store.cc:276] 0807 03:38:57.372593 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:38:57.372752 (+ 159us) service_pool.cc:227] Handling call
0807 03:39:04.730418 (+7357666us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 03:38:58.664388 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 03:38:58.669330 (+ 4942us) write_op.cc:432] Acquiring schema lock in shared mode
0807 03:38:58.669377 (+ 47us) write_op.cc:435] Acquired schema lock
0807 03:38:58.669390 (+ 13us) tablet.cc:660] Decoding operations
0807 03:38:58.670082 (+ 692us) write_op.cc:620] Acquiring the partition lock for write op
0807 03:38:58.670152 (+ 70us) write_op.cc:641] Partition lock acquired for write op
0807 03:38:58.670175 (+ 23us) tablet.cc:683] Acquiring locks for 6 operations
0807 03:38:58.670621 (+ 446us) tablet.cc:699] Row locks acquired
0807 03:38:58.670637 (+ 16us) write_op.cc:260] PREPARE: finished
0807 03:38:58.670760 (+ 123us) write_op.cc:270] Start()
0807 03:38:58.670857 (+ 97us) write_op.cc:276] Timestamp: P: 1786073938670725 usec, L: 0
0807 03:38:58.670874 (+ 17us) op_driver.cc:331] REPLICATION: starting
0807 03:38:58.676316 (+ 5442us) log.cc:842] Serialized 2499 byte log entry
0807 03:38:59.016367 (+340051us) op_driver.cc:444] REPLICATION: finished
0807 03:38:59.275787 (+259420us) write_op.cc:301] APPLY: starting
0807 03:38:59.277973 (+ 2186us) tablet.cc:1365] starting BulkCheckPresence
0807 03:38:59.309603 (+ 31630us) tablet.cc:1368] finished BulkCheckPresence
0807 03:38:59.336619 (+ 27016us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 03:38:59.583255 (+246636us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 03:38:59.657570 (+ 74315us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0807 03:38:59.667045 (+ 9475us) write_op.cc:312] APPLY: finished
0807 03:38:59.871144 (+204099us) log.cc:842] Serialized 52 byte log entry
0807 03:39:00.002467 (+131323us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:39:00.032751 (+ 30284us) write_op.cc:454] Released schema lock
0807 03:39:00.052723 (+ 19972us) write_op.cc:341] FINISH: Updating metrics
0807 03:39:00.095389 (+ 42666us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:39:00.099912 (+ 4523us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1240570,"reported_tablets":19,"child_traces":[["op",{"apply.queue_time_us":234268,"apply.run_cpu_time_us":5312,"apply.run_wall_time_us":865734,"num_ops":6,"prepare.queue_time_us":164282,"prepare.run_cpu_time_us":3833,"prepare.run_wall_time_us":16437,"replication_time_us":345374,"thread_start_us":224637,"threads_started":2}]]}
I20260807 03:39:04.820935 18248 leader_election.cc:302] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:04.752749 20062 raft_consensus.cc:504] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.767861 20159 raft_consensus.cc:482] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:04.778242 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:04.755596 20018 raft_consensus.cc:504] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.826920 18081 raft_consensus.cc:1261] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260807 03:39:04.823410 18080 raft_consensus.cc:2454] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:04.827755 18248 leader_election.cc:302] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:04.821363 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:04.832474 18080 raft_consensus.cc:1261] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260807 03:39:04.832850 18080 pending_rounds.cc:76] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d: Aborting all ops after (but not including) 1
I20260807 03:39:04.833992 18079 raft_consensus.cc:1226] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 1. Current term is 2. Ops: []
I20260807 03:39:04.805632 19659 raft_consensus.cc:729] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:04.833066 18080 pending_rounds.cc:97] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260807 03:39:04.778714 20350 consensus_queue.cc:1046] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.045s
I20260807 03:39:04.752346 18310 rpcz_store.cc:276] 0807 03:39:01.951647 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:02.839544 (+887897us) service_pool.cc:227] Handling call
0807 03:39:04.113539 (+1273995us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:04.166221 (+ 52682us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:04.713484 (+547263us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b540018b0d0. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bea09 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:04.713613 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2926148864}
I20260807 03:39:04.769840 20252 raft_consensus.cc:504] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.872597 18080 raft_consensus.cc:2866] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 03:39:04.762773 20123 raft_consensus.cc:504] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.876626 20137 consensus_queue.cc:1046] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:39:04.752982 20367 consensus_queue.cc:260] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.814863 20208 raft_consensus.cc:2726] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021) with higher term. Message: Invalid argument: T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 4. Current term is 5.
I20260807 03:39:04.872283 20144 raft_consensus.cc:482] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:04.870779 19583 raft_consensus.cc:3037] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
W20260807 03:39:04.887771 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 60078.9us
I20260807 03:39:04.873797 20136 raft_consensus.cc:2726] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 03:39:04.779791 20376 raft_consensus.cc:504] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:04.890693 18081 raft_consensus.cc:1226] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:04.830044 20159 raft_consensus.cc:504] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:04.890870 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 55650.6us
I20260807 03:39:04.877506 20357 raft_consensus.cc:2726] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 03:39:04.877336 19874 raft_consensus.cc:482] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:04.896189 18248 leader_election.cc:302] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:04.881418 20073 raft_consensus.cc:2726] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:04.833994 20091 consensus_queue.cc:1046] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.055s
I20260807 03:39:04.889160 18071 raft_consensus.cc:2379] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 2: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:04.798621 20111 consensus_queue.cc:237] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.951529 20204 leader_election.cc:288] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:04.951205 20275 raft_consensus.cc:504] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.871054 20387 consensus_queue.cc:260] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.952747 20387 raft_consensus.cc:3037] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:04.952862 20252 leader_election.cc:288] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:04.953295 20159 leader_election.cc:288] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:04.954717 18072 consensus_meta.cc:287] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 0.672s user 0.010s sys 0.000s
I20260807 03:39:04.955070 18072 raft_consensus.cc:2454] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 5.
I20260807 03:39:04.955590 20312 raft_consensus.cc:2741] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
W20260807 03:39:04.934713 20411 log.cc:924] Time spent T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20260807 03:39:04.935504 20177 leader_election.cc:288] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:04.958735 18310 raft_consensus.cc:1261] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 03:39:04.962563 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:04.963145 18190 raft_consensus.cc:3037] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:04.965283 20075 raft_consensus.cc:482] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:04.965677 20279 raft_consensus.cc:2726] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 03:39:04.877308 20281 consensus_queue.cc:1046] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20260807 03:39:04.946095 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:04.922827 20144 raft_consensus.cc:504] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.944813 19617 consensus_queue.cc:1046] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
W20260807 03:39:04.949787 20403 log.cc:924] Time spent T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.141s user 0.001s sys 0.000s
I20260807 03:39:04.949477 19874 raft_consensus.cc:504] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.951071 20062 leader_election.cc:288] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:04.973646 18191 raft_consensus.cc:1261] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 03:39:04.941478 18297 raft_consensus.cc:2454] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:39:04.965775 20075 raft_consensus.cc:504] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.990701 20177 raft_consensus.cc:482] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:04.970877 20042 raft_consensus.cc:504] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:04.986577 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:05.031777 18293 raft_consensus.cc:2379] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 4: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:04.998167 20018 leader_election.cc:288] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:04.989728 20159 consensus_queue.cc:1035] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20260807 03:39:04.996570 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:05.035590 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:05.031824 19924 raft_consensus.cc:482] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:05.039690 18293 raft_consensus.cc:1226] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:05.043215 20275 leader_election.cc:288] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:05.044997 19874 leader_election.cc:288] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:05.045148 20389 consensus_queue.cc:1046] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:39:05.048342 20376 leader_election.cc:288] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:05.049510 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:05.051553 20181 consensus_queue.cc:1035] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:39:05.053409 19924 raft_consensus.cc:504] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:04.979346 18190 raft_consensus.cc:2454] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:39:05.012033 19496 leader_election.cc:288] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:05.119726 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:05.115862 18249 leader_election.cc:302] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:05.122757 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:05.123592 18309 raft_consensus.cc:2454] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 2.
I20260807 03:39:05.125623 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:05.134325 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 584059us
I20260807 03:39:05.124888 18027 leader_election.cc:302] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:05.030763 20177 raft_consensus.cc:504] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:05.154732 18140 leader_election.cc:302] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:05.163483 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:05.164351 18081 raft_consensus.cc:2354] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 4. Current term is 5.
I20260807 03:39:05.165159 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:05.167358 18080 raft_consensus.cc:1226] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:05.170174 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:05.170969 18309 raft_consensus.cc:2379] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 4: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
W20260807 03:39:05.132890 20414 log.cc:924] Time spent T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.095s user 0.000s sys 0.001s
I20260807 03:39:05.177214 18309 raft_consensus.cc:1261] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260807 03:39:05.171561 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 46774.1us
I20260807 03:39:05.179150 18027 leader_election.cc:302] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:05.179405 18080 raft_consensus.cc:1226] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 1. Current term is 2. Ops: []
I20260807 03:39:05.180891 18027 leader_election.cc:302] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:05.138718 19496 raft_consensus.cc:2781] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:39:05.139338 20123 leader_election.cc:288] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:05.201458 18180 raft_consensus.cc:1261] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260807 03:39:05.201228 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:05.201795 18180 pending_rounds.cc:76] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02: Aborting all ops after (but not including) 1
I20260807 03:39:05.196414 18027 leader_election.cc:302] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:05.202098 18180 pending_rounds.cc:97] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260807 03:39:05.202369 18180 raft_consensus.cc:2866] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 03:39:05.202695 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:05.152351 20091 raft_consensus.cc:482] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:05.200441 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:05.204398 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:05.204428 18027 leader_election.cc:302] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:05.204725 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:05.205715 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:05.208480 18193 raft_consensus.cc:2379] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 3: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:05.207054 19496 raft_consensus.cc:686] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:05.177443 20275 raft_consensus.cc:2781] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:39:05.209965 19496 consensus_queue.cc:237] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:05.184902 20062 raft_consensus.cc:2781] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:39:05.212677 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:05.184855 20161 raft_consensus.cc:2726] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 03:39:05.215902 18193 raft_consensus.cc:1261] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 03:39:05.226604 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:05.187176 20250 raft_consensus.cc:2726] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 03:39:05.213440 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:39:05.224932 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 328815us
I20260807 03:39:05.228442 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:05.229242 18185 raft_consensus.cc:2454] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 4.
I20260807 03:39:05.231258 18185 raft_consensus.cc:1261] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260807 03:39:05.232131 18185 pending_rounds.cc:76] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02: Aborting all ops after (but not including) 2
I20260807 03:39:05.232837 18185 pending_rounds.cc:97] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260807 03:39:05.206282 20376 raft_consensus.cc:2781] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:39:05.233531 18185 raft_consensus.cc:2866] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 03:39:05.213011 20159 raft_consensus.cc:2781] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader election won for term 4
I20260807 03:39:05.207810 20144 leader_election.cc:288] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:05.237658 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:05.243202 18193 raft_consensus.cc:2454] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 4.
I20260807 03:39:05.242976 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:05.238514 18189 raft_consensus.cc:1261] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 03:39:05.239120 18248 leader_election.cc:302] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:05.224606 20062 raft_consensus.cc:482] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:05.248894 18140 leader_election.cc:398] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Vote denied by peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163) with higher term. Message: Invalid argument: T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 4. Current term is 5.
I20260807 03:39:05.249303 18140 leader_election.cc:401] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260807 03:39:05.231855 20275 raft_consensus.cc:686] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:05.244313 20075 leader_election.cc:288] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:05.233827 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:05.235292 19292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:05.255179 18027 leader_election.cc:302] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:05.259900 18293 raft_consensus.cc:1226] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:05.258590 18027 leader_election.cc:302] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:05.214001 20091 raft_consensus.cc:504] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:05.239904 20018 raft_consensus.cc:482] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:05.251524 20062 raft_consensus.cc:3037] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:05.250649 20243 raft_consensus.cc:2781] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 03:39:05.242588 20376 raft_consensus.cc:686] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:05.253530 20389 raft_consensus.cc:3037] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:05.259167 18140 leader_election.cc:302] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:05.248245 20159 raft_consensus.cc:686] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:05.264503 18027 leader_election.cc:302] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:05.255955 20042 leader_election.cc:288] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:05.256660 20425 raft_consensus.cc:482] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:05.267582 20161 raft_consensus.cc:2726] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 03:39:05.282747 18079 raft_consensus.cc:1261] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:39:05.260741 20111 consensus_queue.cc:1046] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.013s
I20260807 03:39:05.295658 19295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:05.260181 19874 consensus_queue.cc:1035] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20260807 03:39:05.305599 18248 leader_election.cc:302] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:05.271153 20399 raft_consensus.cc:2726] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 03:39:05.266491 20154 consensus_queue.cc:1046] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.030s
I20260807 03:39:05.271086 20208 consensus_queue.cc:1046] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
W20260807 03:39:05.309338 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 45044.1us
I20260807 03:39:05.297252 20042 raft_consensus.cc:2726] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:05.268548 20018 raft_consensus.cc:504] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:05.329085 18248 leader_election.cc:302] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:05.329867 18027 leader_election.cc:302] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:05.296265 20376 consensus_queue.cc:237] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:05.272176 19924 leader_election.cc:288] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:05.272399 20243 raft_consensus.cc:686] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:05.376065 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:05.396692 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 66919us
I20260807 03:39:05.402954 18027 leader_election.cc:302] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:05.347903 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:05.334013 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:05.408393 18081 raft_consensus.cc:1226] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 1. Current term is 2. Ops: []
W20260807 03:39:05.332502 20219 consensus_meta.cc:287] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.517s user 0.009s sys 0.000s
I20260807 03:39:05.308880 20357 consensus_queue.cc:1046] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.104s
I20260807 03:39:05.412804 18027 leader_election.cc:302] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:05.417325 20252 raft_consensus.cc:2726] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:05.403817 20042 raft_consensus.cc:3037] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:05.272405 20181 consensus_queue.cc:1035] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.140s
I20260807 03:39:05.420609 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:05.428850 18070 raft_consensus.cc:2379] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 4: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
W20260807 03:39:05.333981 20124 consensus_meta.cc:287] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 0.510s user 0.008s sys 0.002s
W20260807 03:39:05.421417 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 67806.9us
I20260807 03:39:05.297969 20425 raft_consensus.cc:504] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:05.329555 20434 raft_consensus.cc:482] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:05.327531 20279 raft_consensus.cc:2726] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 03:39:05.312721 20177 leader_election.cc:288] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:05.424964 18027 leader_election.cc:302] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:05.308733 20159 consensus_queue.cc:237] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:05.493144 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 68291.4us
I20260807 03:39:05.493424 20365 raft_consensus.cc:3032] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:39:05.493803 20365 raft_consensus.cc:729] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:05.494091 18027 leader_election.cc:302] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:05.328984 20150 consensus_queue.cc:1035] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.093s
I20260807 03:39:05.494570 20365 consensus_queue.cc:260] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:05.495327 20252 raft_consensus.cc:2726] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:05.495968 20252 raft_consensus.cc:2781] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:05.400899 20433 raft_consensus.cc:2726] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 03:39:05.333161 20075 raft_consensus.cc:2726] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 03:39:05.496964 18027 leader_election.cc:302] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:05.497292 18079 raft_consensus.cc:2379] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 3: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:05.434126 18081 raft_consensus.cc:1261] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260807 03:39:05.497718 20250 raft_consensus.cc:2726] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:05.498642 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:05.499120 20250 consensus_queue.cc:1035] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:39:05.500895 18027 leader_election.cc:302] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:05.501672 20250 raft_consensus.cc:2781] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:05.502056 20250 raft_consensus.cc:482] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:05.496960 20433 consensus_queue.cc:1035] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20260807 03:39:05.502524 20250 raft_consensus.cc:3037] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:05.503836 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:05.504529 18300 raft_consensus.cc:2454] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 2.
I20260807 03:39:05.434944 20018 leader_election.cc:288] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:05.435708 20430 log.cc:924] Time spent T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.105s user 0.000s sys 0.000s
W20260807 03:39:05.506098 20416 log.cc:924] Time spent T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.492s user 0.010s sys 0.000s
I20260807 03:39:05.481313 20311 ts_tablet_manager.cc:1918] T af8877c3fbba4ab18765832f1390921f P 66a36e1ac8e945e893668e282d394c49: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 03:39:05.436167 20091 leader_election.cc:288] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
W20260807 03:39:05.436331 20427 log.cc:924] Time spent T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.164s user 0.001s sys 0.000s
I20260807 03:39:05.435842 20243 consensus_queue.cc:237] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:05.508970 20062 raft_consensus.cc:504] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
W20260807 03:39:05.438923 20173 consensus_meta.cc:287] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 0.800s user 0.007s sys 0.002s
I20260807 03:39:05.307425 20275 consensus_queue.cc:237] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:05.512245 18080 raft_consensus.cc:2454] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
I20260807 03:39:05.497771 20277 raft_consensus.cc:3032] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:39:05.512763 20277 raft_consensus.cc:729] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:05.429080 20441 raft_consensus.cc:2726] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 03:39:05.442464 20425 leader_election.cc:288] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:05.443239 19292 raft_consensus.cc:2379] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 2: Already voted for candidate 5c9b5062bdfa46af9396f1a85025a331 in this term.
I20260807 03:39:05.446175 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:05.423681 20219 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:05.331010 20137 consensus_queue.cc:1046] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.096s
I20260807 03:39:05.455499 18187 raft_consensus.cc:1261] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:39:05.485069 18293 raft_consensus.cc:1226] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:05.490986 19659 consensus_queue.cc:260] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:05.309264 19583 consensus_queue.cc:1035] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.118s
I20260807 03:39:05.496366 20252 raft_consensus.cc:482] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:05.447911 20389 raft_consensus.cc:2726] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163) with higher term. Message: Invalid argument: T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 4. Current term is 5.
I20260807 03:39:05.511864 20311 log.cc:1196] T af8877c3fbba4ab18765832f1390921f P 66a36e1ac8e945e893668e282d394c49: Deleting WAL directory at /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786073845599630-16154-0/minicluster-data/ts-2-root/wals/af8877c3fbba4ab18765832f1390921f
I20260807 03:39:05.492334 20434 raft_consensus.cc:504] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:05.566036 20311 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57260 (request call id 207) took 2743 ms. Trace:
I20260807 03:39:05.566429 20311 rpcz_store.cc:276] 0807 03:39:02.822252 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:02.826928 (+ 4676us) service_pool.cc:227] Handling call
0807 03:39:03.074584 (+247656us) ts_tablet_manager.cc:1145] Deleting tablet af8877c3fbba4ab18765832f1390921f
0807 03:39:03.110172 (+ 35588us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0807 03:39:04.787126 (+1676954us) tablet_metadata.cc:641] Metadata flushed
0807 03:39:05.458365 (+671239us) tablet_metadata.cc:641] Metadata flushed
0807 03:39:05.565941 (+107576us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1165018,"tablet-delete.queue_time_us":227616,"thread_start_us":112553,"threads_started":1}
I20260807 03:39:05.567679 20434 leader_election.cc:288] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:05.517484 18027 leader_election.cc:302] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:05.513568 20277 consensus_queue.cc:260] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:05.544305 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 54055.7us
I20260807 03:39:05.569674 20219 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:05.501691 20076 raft_consensus.cc:3032] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:39:05.570719 20137 raft_consensus.cc:3032] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:39:05.571157 20137 raft_consensus.cc:729] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:05.571969 20137 consensus_queue.cc:260] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:05.561667 18082 raft_consensus.cc:1261] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:39:05.561017 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:05.570729 20076 raft_consensus.cc:729] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:05.575091 17899 catalog_manager.cc:4841] TS 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021): tablet af8877c3fbba4ab18765832f1390921f (table test-workload [id=677ca051b50345e183d60556fc42291d]) successfully deleted
I20260807 03:39:05.575709 18082 raft_consensus.cc:1261] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 03:39:05.547209 20111 consensus_queue.cc:1035] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20260807 03:39:05.576135 20076 consensus_queue.cc:260] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:05.534054 18079 raft_consensus.cc:2932] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:05.577672 20076 raft_consensus.cc:3037] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:05.578274 18312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57268 (request call id 920) took 1154 ms. Trace:
I20260807 03:39:05.578648 18312 rpcz_store.cc:276] 0807 03:39:04.423680 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:04.426211 (+ 2531us) service_pool.cc:227] Handling call
0807 03:39:05.578110 (+1151899us) spinlock_profiling.cc:243] Waited 342 ms on lock 0x7b54002b48d4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:05.578221 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":786744576}
I20260807 03:39:05.553341 20275 raft_consensus.cc:482] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:05.580309 18184 raft_consensus.cc:2454] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 5.
I20260807 03:39:05.580576 20275 raft_consensus.cc:504] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:05.582849 20275 leader_election.cc:288] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:05.586719 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:05.587632 18072 raft_consensus.cc:2379] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 2: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:05.528647 20018 consensus_queue.cc:1035] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.060s
I20260807 03:39:05.591352 18293 raft_consensus.cc:1261] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:39:05.587467 20250 raft_consensus.cc:504] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:05.592523 18141 leader_election.cc:398] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Vote denied by peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021) with higher term. Message: Invalid argument: T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3.
I20260807 03:39:05.592703 20062 leader_election.cc:288] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Requested vote from peers 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
W20260807 03:39:05.595009 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 77607us
I20260807 03:39:05.595528 20434 raft_consensus.cc:2726] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:05.595767 19297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331"
I20260807 03:39:05.597421 18248 leader_election.cc:302] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:05.598105 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:05.602407 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:05.603050 18312 raft_consensus.cc:2454] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 5.
I20260807 03:39:05.600878 18141 leader_election.cc:401] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260807 03:39:05.562889 20252 raft_consensus.cc:3037] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
W20260807 03:39:05.605809 20398 log.cc:924] Time spent T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 1.077s user 0.002s sys 0.013s
I20260807 03:39:05.606336 20042 raft_consensus.cc:2726] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:05.606825 20073 raft_consensus.cc:3032] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:39:05.607224 20073 raft_consensus.cc:729] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:05.608296 18081 raft_consensus.cc:2454] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
I20260807 03:39:05.610340 18141 leader_election.cc:302] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:05.549775 20363 raft_consensus.cc:482] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:05.612808 18312 raft_consensus.cc:1226] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 4. Current term is 5. Ops: []
I20260807 03:39:05.621165 18248 leader_election.cc:302] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:39:05.625105 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:05.625690 18293 raft_consensus.cc:3037] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:05.625972 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:05.627405 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:05.628443 18300 raft_consensus.cc:2379] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 2: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:05.614470 20204 raft_consensus.cc:2726] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 03:39:05.611301 20233 raft_consensus.cc:2726] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021) with higher term. Message: Invalid argument: T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3.
I20260807 03:39:05.616492 20275 raft_consensus.cc:2781] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:39:05.621601 20150 consensus_queue.cc:1035] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20260807 03:39:05.632555 19240 leader_election.cc:302] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:05.608296 20073 consensus_queue.cc:260] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:05.633711 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:05.634330 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:05.621560 20180 consensus_queue.cc:1035] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.013s
I20260807 03:39:05.639400 18293 raft_consensus.cc:2454] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 5.
I20260807 03:39:05.643003 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:05.628861 20243 raft_consensus.cc:2781] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:39:05.646162 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:05.647634 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:05.648027 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:05.615712 20387 consensus_queue.cc:1035] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20260807 03:39:05.650529 18027 leader_election.cc:302] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:05.650959 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:05.649895 18185 raft_consensus.cc:1226] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 2. Current term is 3. Ops: []
I20260807 03:39:05.656420 18082 raft_consensus.cc:2454] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 5.
I20260807 03:39:05.659207 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:05.615401 20076 consensus_queue.cc:1046] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20260807 03:39:05.619247 20389 consensus_queue.cc:1046] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20260807 03:39:05.659356 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:05.617110 20363 raft_consensus.cc:504] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:05.659942 18193 raft_consensus.cc:3037] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:05.660002 20250 leader_election.cc:288] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:05.660982 20425 raft_consensus.cc:2781] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:39:05.630424 20446 consensus_queue.cc:1035] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.038s
I20260807 03:39:05.641981 20275 raft_consensus.cc:482] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:05.649768 20042 raft_consensus.cc:482] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:05.665091 20073 raft_consensus.cc:3037] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:05.657904 20243 raft_consensus.cc:482] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:05.621527 20159 consensus_queue.cc:1035] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.037s
I20260807 03:39:05.675922 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
W20260807 03:39:05.716502 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 66020.2us
I20260807 03:39:05.665015 18140 leader_election.cc:302] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:05.640911 20111 raft_consensus.cc:482] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:05.649962 20173 raft_consensus.cc:482] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:05.876677 20425 raft_consensus.cc:482] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 03:39:05.879380 20456 log.cc:924] Time spent T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.330s user 0.008s sys 0.017s
I20260807 03:39:05.634749 20265 consensus_queue.cc:260] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:05.897311 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:05.879873 20463 log.cc:924] Time spent T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.232s user 0.006s sys 0.000s
I20260807 03:39:05.886065 18300 raft_consensus.cc:1226] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:05.906136 18193 raft_consensus.cc:2454] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 5.
I20260807 03:39:05.906608 18027 leader_election.cc:302] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
W20260807 03:39:05.914738 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 249770us
I20260807 03:39:05.915305 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:05.886932 20275 raft_consensus.cc:3037] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:05.899238 20243 raft_consensus.cc:3037] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:05.913038 20156 raft_consensus.cc:3032] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:39:05.906616 20111 raft_consensus.cc:504] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:05.955155 20265 raft_consensus.cc:3037] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
W20260807 03:39:05.955514 20471 log.cc:924] Time spent T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.290s user 0.001s sys 0.000s
I20260807 03:39:05.968967 20111 leader_election.cc:288] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:05.881137 19874 raft_consensus.cc:2726] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:05.907262 20425 raft_consensus.cc:3037] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:05.908550 20365 raft_consensus.cc:3037] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:05.964848 18140 leader_election.cc:302] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:05.957865 20250 raft_consensus.cc:2781] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election won for term 5
I20260807 03:39:05.906167 20441 raft_consensus.cc:482] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:05.905761 20173 raft_consensus.cc:504] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:05.961972 20156 raft_consensus.cc:729] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:05.914095 20312 raft_consensus.cc:482] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:06.007704 20395 raft_consensus.cc:3037] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:05.898010 20042 raft_consensus.cc:504] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.012545 20233 raft_consensus.cc:482] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:06.019559 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:05.977536 20111 raft_consensus.cc:482] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:05.959929 18027 leader_election.cc:302] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:06.039891 18305 raft_consensus.cc:2454] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:39:06.043169 18141 leader_election.cc:302] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:05.987924 19704 raft_consensus.cc:482] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
W20260807 03:39:06.046118 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 86288.6us
I20260807 03:39:06.022197 20250 raft_consensus.cc:686] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:06.034607 20437 raft_consensus.cc:482] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:06.060163 18312 raft_consensus.cc:1261] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:39:06.044157 20387 raft_consensus.cc:482] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:06.025547 20441 raft_consensus.cc:504] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.055732 20433 raft_consensus.cc:2726] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:06.073081 19874 raft_consensus.cc:482] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:06.108677 18075 raft_consensus.cc:1261] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260807 03:39:06.111238 20460 log.cc:924] Time spent T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331: Append to log took a long time: real 0.569s user 0.003s sys 0.000s
I20260807 03:39:06.047765 20350 raft_consensus.cc:482] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:06.076706 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:06.023926 20312 raft_consensus.cc:504] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.107517 19861 raft_consensus.cc:2781] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260807 03:39:06.116488 20415 log.cc:924] Time spent T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 1.103s user 0.000s sys 0.014s
I20260807 03:39:06.057806 20436 consensus_queue.cc:1046] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.054s
I20260807 03:39:06.132030 19583 raft_consensus.cc:2726] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:06.044513 20233 raft_consensus.cc:504] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.091387 20387 raft_consensus.cc:504] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.091511 20437 raft_consensus.cc:504] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.134814 19874 raft_consensus.cc:504] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.157243 18082 raft_consensus.cc:1226] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 1. Current term is 2. Ops: []
I20260807 03:39:06.157487 20173 leader_election.cc:288] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:06.158564 20252 consensus_meta.cc:287] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 0.553s user 0.010s sys 0.000s
I20260807 03:39:06.153749 20250 consensus_queue.cc:237] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.048228 20111 raft_consensus.cc:504] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:06.163813 18073 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33268 (request call id 880) took 2728 ms (client timeout 1961 ms). Trace:
I20260807 03:39:06.132679 20423 raft_consensus.cc:482] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:06.068749 19704 raft_consensus.cc:504] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.164342 18072 raft_consensus.cc:2454] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:39:06.172794 20042 leader_election.cc:288] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:06.119017 17920 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35098 (request call id 23) took 3290 ms. Trace:
I20260807 03:39:06.134430 20363 leader_election.cc:288] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:06.170627 19861 raft_consensus.cc:482] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:06.204511 18312 raft_consensus.cc:1261] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260807 03:39:06.204166 18248 leader_election.cc:302] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
W20260807 03:39:06.205984 18073 rpcz_store.cc:269] 0807 03:39:03.431245 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:03.499366 (+ 68121us) service_pool.cc:227] Handling call
0807 03:39:06.159332 (+2659966us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b54002c9054. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:06.159473 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3054419200}
I20260807 03:39:06.208389 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:06.213440 18310 raft_consensus.cc:1226] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 1. Current term is 2. Ops: []
I20260807 03:39:06.090966 20156 consensus_queue.cc:260] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.202885 20136 raft_consensus.cc:482] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:06.200636 20389 raft_consensus.cc:482] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:06.159246 20350 raft_consensus.cc:504] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.240320 18190 raft_consensus.cc:1226] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260807 03:39:06.230429 17920 rpcz_store.cc:276] 0807 03:39:02.828832 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:02.863378 (+ 34546us) service_pool.cc:227] Handling call
0807 03:39:06.118949 (+3255571us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 03:39:05.145603 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 03:39:05.146223 (+ 620us) write_op.cc:432] Acquiring schema lock in shared mode
0807 03:39:05.146265 (+ 42us) write_op.cc:435] Acquired schema lock
0807 03:39:05.146293 (+ 28us) tablet.cc:660] Decoding operations
0807 03:39:05.146819 (+ 526us) write_op.cc:620] Acquiring the partition lock for write op
0807 03:39:05.146905 (+ 86us) write_op.cc:641] Partition lock acquired for write op
0807 03:39:05.146942 (+ 37us) tablet.cc:683] Acquiring locks for 2 operations
0807 03:39:05.147327 (+ 385us) tablet.cc:699] Row locks acquired
0807 03:39:05.147346 (+ 19us) write_op.cc:260] PREPARE: finished
0807 03:39:05.147533 (+ 187us) write_op.cc:270] Start()
0807 03:39:05.147650 (+ 117us) write_op.cc:276] Timestamp: P: 1786073945147485 usec, L: 0
0807 03:39:05.147677 (+ 27us) op_driver.cc:331] REPLICATION: starting
0807 03:39:05.236916 (+ 89239us) log.cc:842] Serialized 890 byte log entry
0807 03:39:05.482239 (+245323us) op_driver.cc:444] REPLICATION: finished
0807 03:39:05.544841 (+ 62602us) write_op.cc:301] APPLY: starting
0807 03:39:05.544953 (+ 112us) tablet.cc:1365] starting BulkCheckPresence
0807 03:39:05.545114 (+ 161us) tablet.cc:1368] finished BulkCheckPresence
0807 03:39:05.545294 (+ 180us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 03:39:05.612866 (+ 67572us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 03:39:05.623186 (+ 10320us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0807 03:39:05.630501 (+ 7315us) write_op.cc:312] APPLY: finished
0807 03:39:05.655577 (+ 25076us) log.cc:842] Serialized 28 byte log entry
0807 03:39:05.668396 (+ 12819us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:39:05.876147 (+207751us) write_op.cc:454] Released schema lock
0807 03:39:05.880305 (+ 4158us) write_op.cc:341] FINISH: Updating metrics
0807 03:39:05.905437 (+ 25132us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:39:05.907751 (+ 2314us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1470507,"reported_tablets":2,"child_traces":[["op",{"apply.queue_time_us":62295,"apply.run_cpu_time_us":14689,"apply.run_wall_time_us":406615,"num_ops":2,"prepare.queue_time_us":306807,"prepare.run_cpu_time_us":90152,"prepare.run_wall_time_us":249939,"raft.queue_time_us":16624,"raft.run_cpu_time_us":4155,"raft.run_wall_time_us":71001,"replication_time_us":334431,"thread_start_us":463356,"threads_started":4,"wal-append.queue_time_us":174354}]]}
I20260807 03:39:06.246683 18073 raft_consensus.cc:1261] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260807 03:39:06.237082 19861 raft_consensus.cc:3037] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:06.248479 20149 raft_consensus.cc:2781] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:39:06.244560 20205 raft_consensus.cc:482] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:06.209054 20252 raft_consensus.cc:504] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.230820 20423 raft_consensus.cc:504] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.271788 20159 raft_consensus.cc:482] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 03:39:06.308297 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 53094.7us
W20260807 03:39:06.309957 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 72668.1us
I20260807 03:39:06.282742 20149 raft_consensus.cc:482] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:06.249203 20136 raft_consensus.cc:504] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.249346 20389 raft_consensus.cc:504] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.327445 18027 leader_election.cc:302] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:06.309854 20042 raft_consensus.cc:482] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:06.299558 20205 raft_consensus.cc:504] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.318069 20487 raft_consensus.cc:482] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:06.327647 20149 raft_consensus.cc:3037] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
W20260807 03:39:06.359946 20478 log.cc:924] Time spent T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.274s user 0.001s sys 0.000s
I20260807 03:39:06.388823 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:06.329209 20490 raft_consensus.cc:482] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:06.393025 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:06.393695 18073 raft_consensus.cc:2454] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 6.
I20260807 03:39:06.394658 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:06.395534 18073 raft_consensus.cc:2379] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 4: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
W20260807 03:39:06.401434 18141 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.15.198.131:33021 timed out after 1.980s (SENT) [suppressed 9 similar messages]
I20260807 03:39:06.372196 20437 leader_election.cc:288] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:06.317073 20159 raft_consensus.cc:504] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.431927 20493 raft_consensus.cc:482] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:06.400656 20312 leader_election.cc:288] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:06.414906 20387 leader_election.cc:288] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:06.453989 20493 raft_consensus.cc:504] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.389715 20487 raft_consensus.cc:504] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.456875 20493 leader_election.cc:288] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:06.432206 20437 raft_consensus.cc:482] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:06.457729 20487 leader_election.cc:288] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:06.397622 20446 raft_consensus.cc:482] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:06.459455 20446 raft_consensus.cc:504] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.462616 20446 leader_election.cc:288] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:06.389485 20436 raft_consensus.cc:3037] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:06.422763 20233 leader_election.cc:288] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:06.374104 20042 raft_consensus.cc:504] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:06.422192 20503 consensus_queue.cc:1035] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.097s
W20260807 03:39:06.506589 20481 log.cc:924] Time spent T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.278s user 0.001s sys 0.000s
I20260807 03:39:06.436707 20500 raft_consensus.cc:482] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:06.518770 20500 raft_consensus.cc:504] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:06.521030 20501 log.cc:924] Time spent T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.107s user 0.000s sys 0.000s
I20260807 03:39:06.521816 20500 leader_election.cc:288] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:06.506346 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 178955us
I20260807 03:39:06.460734 20437 raft_consensus.cc:504] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.444693 20111 leader_election.cc:288] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:06.515341 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:06.481971 19704 leader_election.cc:288] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:06.505900 20487 raft_consensus.cc:482] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:06.473726 20281 raft_consensus.cc:482] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:06.484448 20495 raft_consensus.cc:482] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:06.402581 20490 raft_consensus.cc:504] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.507725 20350 leader_election.cc:288] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:06.556957 20495 raft_consensus.cc:504] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.558497 20490 leader_election.cc:288] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:06.531165 20423 leader_election.cc:288] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
W20260807 03:39:06.561362 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 159963us
I20260807 03:39:06.542721 20219 consensus_queue.cc:1035] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20260807 03:39:06.528573 20498 raft_consensus.cc:482] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:06.533274 20500 raft_consensus.cc:482] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:06.565001 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:06.563019 18027 leader_election.cc:302] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:06.571991 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:06.572338 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:06.543215 20503 raft_consensus.cc:482] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:06.565012 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:06.548278 20252 leader_election.cc:288] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:06.588255 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:06.568272 20265 consensus_meta.cc:287] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 0.601s user 0.012s sys 0.000s
I20260807 03:39:06.596236 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:06.596403 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:06.597004 18312 raft_consensus.cc:2454] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:39:06.591499 18310 raft_consensus.cc:2379] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 6: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:06.598062 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:06.592864 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:06.598713 18312 raft_consensus.cc:2454] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 6.
I20260807 03:39:06.600821 18141 leader_election.cc:302] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:06.589584 20136 leader_election.cc:288] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
W20260807 03:39:06.608919 20485 log.cc:924] Time spent T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.271s user 0.000s sys 0.001s
I20260807 03:39:06.609488 20159 leader_election.cc:288] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:06.562302 20517 raft_consensus.cc:482] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:06.612917 18079 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33254 (request call id 1035) took 1123 ms. Trace:
I20260807 03:39:06.523522 20512 consensus_queue.cc:1046] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.110s
I20260807 03:39:06.555927 20487 raft_consensus.cc:504] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.559733 20281 raft_consensus.cc:504] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.585891 20498 raft_consensus.cc:504] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.590164 20350 raft_consensus.cc:482] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:06.589326 20495 leader_election.cc:288] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:06.589453 20490 raft_consensus.cc:482] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
W20260807 03:39:06.683089 20499 log.cc:924] Time spent T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.145s user 0.000s sys 0.001s
I20260807 03:39:06.600986 20446 raft_consensus.cc:482] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:06.633153 20515 consensus_queue.cc:1046] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.172s
I20260807 03:39:06.761554 18309 raft_consensus.cc:2454] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:39:06.703931 20154 raft_consensus.cc:3032] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Stepping down as leader of term 4
W20260807 03:39:06.703733 20425 consensus_meta.cc:287] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331: Time spent flushing consensus metadata: real 0.691s user 0.007s sys 0.004s
I20260807 03:39:06.689358 20507 raft_consensus.cc:482] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:06.763135 20495 raft_consensus.cc:3032] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 LEADER]: Stepping down as leader of term 2
I20260807 03:39:06.763856 20495 raft_consensus.cc:729] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:06.693600 20437 leader_election.cc:288] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:06.762252 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:06.763032 20425 raft_consensus.cc:504] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:06.764380 20487 leader_election.cc:288] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:06.668527 20504 raft_consensus.cc:482] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:06.764837 18072 raft_consensus.cc:2454] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 6.
I20260807 03:39:06.646945 20389 leader_election.cc:288] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:06.765146 20504 raft_consensus.cc:504] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.765605 20517 raft_consensus.cc:504] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:06.595351 20503 raft_consensus.cc:504] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.781049 18248 leader_election.cc:302] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:06.766347 20495 consensus_queue.cc:260] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.589586 20500 raft_consensus.cc:504] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:06.668547 20505 log.cc:924] Time spent T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
W20260807 03:39:06.695308 20243 consensus_meta.cc:287] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.719s user 0.005s sys 0.007s
I20260807 03:39:06.787438 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:06.702548 18168 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b214306f53244f8b6ff2a1a9019abd1 with delete_type TABLET_DATA_TOMBSTONED (TS 3c4bceb836db4536a0f255fdf86abc02 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44650
I20260807 03:39:06.758255 20350 raft_consensus.cc:504] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.612459 20111 raft_consensus.cc:482] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:06.759898 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:06.684466 20387 raft_consensus.cc:482] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:06.768827 20504 leader_election.cc:288] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:06.715950 20521 raft_consensus.cc:482] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:06.595364 20205 leader_election.cc:288] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
W20260807 03:39:06.694097 20073 consensus_meta.cc:287] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.796s user 0.009s sys 0.002s
I20260807 03:39:06.763456 20507 raft_consensus.cc:504] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:06.720396 19292 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46882 (request call id 12) took 1609 ms (client timeout 2082 ms). Trace:
I20260807 03:39:06.756207 18079 rpcz_store.cc:276] 0807 03:39:05.489419 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:05.509270 (+ 19851us) service_pool.cc:227] Handling call
0807 03:39:05.513180 (+ 3910us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 03:39:05.533742 (+ 20562us) raft_consensus.cc:1520] Early marking committed up to index 3
0807 03:39:05.612238 (+ 78496us) log.cc:842] Serialized 14 byte log entry
0807 03:39:06.162303 (+550065us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 03:39:06.194546 (+ 32243us) log.cc:842] Serialized 52 byte log entry
0807 03:39:06.199400 (+ 4854us) raft_consensus.cc:1631] Marking committed up to 3
0807 03:39:06.199438 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:06.199529 (+ 91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:06.612622 (+413093us) raft_consensus.cc:1671] finished
0807 03:39:06.612651 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:06.612827 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21383,"raft.queue_time_us":86080,"raft.run_cpu_time_us":290,"raft.run_wall_time_us":34651,"thread_start_us":492968,"threads_started":2,"wal-append.queue_time_us":599381}
I20260807 03:39:06.772238 20446 raft_consensus.cc:504] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.768496 20425 leader_election.cc:288] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:06.773313 19861 raft_consensus.cc:504] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.611553 20042 leader_election.cc:288] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:06.720484 20532 raft_consensus.cc:482] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
W20260807 03:39:06.775326 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 189004us
W20260807 03:39:06.690165 20365 consensus_meta.cc:287] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 0.681s user 0.008s sys 0.001s
W20260807 03:39:06.820384 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 219626us
I20260807 03:39:06.794533 20528 raft_consensus.cc:482] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:06.668022 19295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:06.767853 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:06.793573 20243 raft_consensus.cc:504] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:06.793262 18073 raft_consensus.cc:2454] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:06.804019 20387 raft_consensus.cc:504] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.776407 20159 raft_consensus.cc:2726] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 03:39:06.784718 20423 raft_consensus.cc:482] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:06.794606 20347 raft_consensus.cc:3032] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:39:06.667699 18177 tablet_service.cc:1514] Processing DeleteTablet for tablet 071a939ead594d38851e3b554bfc4d24 with delete_type TABLET_DATA_TOMBSTONED (TS 3c4bceb836db4536a0f255fdf86abc02 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44650
I20260807 03:39:06.812711 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:06.813570 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:06.860210 18079 raft_consensus.cc:2379] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 4: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:06.795311 20389 raft_consensus.cc:2781] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:06.796972 20498 leader_election.cc:288] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:06.749421 20149 raft_consensus.cc:504] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.782176 20490 raft_consensus.cc:504] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.768777 20154 raft_consensus.cc:729] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:06.811761 20205 raft_consensus.cc:2781] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:39:06.830917 18189 raft_consensus.cc:1261] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:39:06.830953 18141 leader_election.cc:302] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:06.868701 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:06.833266 18027 leader_election.cc:302] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:06.869783 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:06.870949 18027 leader_election.cc:302] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:06.834627 20547 raft_consensus.cc:482] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:06.835296 20365 raft_consensus.cc:482] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:06.843585 20532 raft_consensus.cc:504] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:06.836344 19861 leader_election.cc:288] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:06.814371 20521 raft_consensus.cc:504] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.762559 20281 leader_election.cc:288] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:06.638510 20265 raft_consensus.cc:482] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:06.851101 18248 leader_election.cc:302] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
W20260807 03:39:06.847317 20275 consensus_meta.cc:287] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 0.882s user 0.009s sys 0.003s
I20260807 03:39:06.861663 18295 raft_consensus.cc:2354] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 2. Current term is 3.
I20260807 03:39:06.849982 20436 raft_consensus.cc:2726] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:06.798623 20111 raft_consensus.cc:504] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:06.852321 18072 raft_consensus.cc:3037] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:06.847069 20512 raft_consensus.cc:482] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:06.846604 19295 raft_consensus.cc:2454] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 2.
I20260807 03:39:06.873633 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:06.811338 19292 rpcz_store.cc:269] 0807 03:39:05.142208 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:05.184565 (+ 42357us) service_pool.cc:227] Handling call
0807 03:39:06.718552 (+1533987us) spinlock_profiling.cc:243] Waited 183 ms on lock 0x7b54006103d4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:06.720303 (+ 1751us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":420047104}
I20260807 03:39:06.844726 20528 raft_consensus.cc:504] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.859577 20347 raft_consensus.cc:729] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:06.918264 20347 consensus_queue.cc:260] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.863216 20389 raft_consensus.cc:482] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:06.886606 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:06.919970 20389 raft_consensus.cc:3037] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:06.920964 20446 leader_election.cc:288] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:06.922216 20537 raft_consensus.cc:2726] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 03:39:06.874557 20205 raft_consensus.cc:482] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:06.895391 20542 raft_consensus.cc:482] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:06.896677 17920 catalog_manager.cc:5495] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:06.898940 18187 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44660 (request call id 1142) took 1911 ms. Trace:
I20260807 03:39:06.901965 18069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33254 (request call id 1017) took 2135 ms. Trace:
I20260807 03:39:06.911580 18312 raft_consensus.cc:2454] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
W20260807 03:39:06.911082 20395 consensus_meta.cc:287] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 0.887s user 0.013s sys 0.000s
I20260807 03:39:06.863394 20423 raft_consensus.cc:504] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.896791 20275 raft_consensus.cc:504] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.917127 20528 leader_election.cc:288] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:06.945714 18069 rpcz_store.cc:276] 0807 03:39:04.766089 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:04.831245 (+ 65156us) service_pool.cc:227] Handling call
0807 03:39:04.881042 (+ 49797us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:05.007979 (+126937us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:05.008014 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:05.008059 (+ 45us) raft_consensus.cc:1631] Marking committed up to 2
0807 03:39:05.008120 (+ 61us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:06.901721 (+1893601us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x7b540037df54. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:39:06.901767 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:06.901896 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":291419008}
I20260807 03:39:06.925429 20149 leader_election.cc:288] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:06.925988 18025 leader_election.cc:302] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:06.880208 20365 raft_consensus.cc:504] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.894263 20504 raft_consensus.cc:2781] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:39:06.934172 18187 rpcz_store.cc:276] 0807 03:39:04.987394 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:04.987521 (+ 127us) service_pool.cc:227] Handling call
0807 03:39:04.987823 (+ 302us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:05.546655 (+558832us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:06.898716 (+1352061us) spinlock_profiling.cc:243] Waited 463 ms on lock 0x7b54002bedd4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:39:06.898888 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1064763776}
I20260807 03:39:06.944272 20243 leader_election.cc:288] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:06.907560 20281 raft_consensus.cc:2781] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:39:06.897675 20265 raft_consensus.cc:504] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.937299 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:06.968437 18312 raft_consensus.cc:3037] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
W20260807 03:39:06.937165 20422 log.cc:924] Time spent T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 1.668s user 0.008s sys 0.000s
I20260807 03:39:06.948392 18141 leader_election.cc:302] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:06.915364 20512 raft_consensus.cc:504] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.973858 18183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44660 (request call id 1131) took 2141 ms. Trace:
I20260807 03:39:06.880352 20547 raft_consensus.cc:504] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.936820 20551 raft_consensus.cc:482] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:06.943218 20500 leader_election.cc:288] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:06.943377 20395 raft_consensus.cc:2726] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:06.933370 20205 raft_consensus.cc:3037] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:06.975458 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:06.946671 20503 leader_election.cc:288] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:06.988374 19292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331"
I20260807 03:39:06.988605 20275 leader_election.cc:288] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:06.953279 20498 raft_consensus.cc:2781] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:39:06.985342 18187 raft_consensus.cc:2454] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 2.
I20260807 03:39:06.956629 20504 raft_consensus.cc:482] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:06.953116 20436 raft_consensus.cc:482] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:06.956672 20281 raft_consensus.cc:482] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:06.986574 20387 leader_election.cc:288] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:06.986984 20532 leader_election.cc:288] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:06.987839 20417 raft_consensus.cc:2781] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election won for term 5
I20260807 03:39:07.001075 20417 raft_consensus.cc:686] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:07.001313 20521 leader_election.cc:288] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:06.970232 20042 raft_consensus.cc:482] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:06.989236 19292 raft_consensus.cc:2379] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 3: Already voted for candidate 5c9b5062bdfa46af9396f1a85025a331 in this term.
I20260807 03:39:06.970484 19861 raft_consensus.cc:482] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:06.988997 20507 leader_election.cc:288] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:06.975162 20437 consensus_queue.cc:1035] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20260807 03:39:06.976506 20517 leader_election.cc:288] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:06.997164 20500 raft_consensus.cc:2781] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:39:06.990875 20503 raft_consensus.cc:482] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:07.013587 20512 leader_election.cc:288] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:06.991560 20498 raft_consensus.cc:482] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:07.000408 20504 raft_consensus.cc:3037] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:06.990191 20541 raft_consensus.cc:482] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:07.003343 20281 raft_consensus.cc:3037] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:07.020277 20503 raft_consensus.cc:504] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:07.006031 20365 leader_election.cc:288] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:07.098675 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:07.008785 20042 raft_consensus.cc:504] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:07.145856 20537 raft_consensus.cc:482] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:07.147377 20347 raft_consensus.cc:3037] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:06.974225 18183 rpcz_store.cc:276] 0807 03:39:04.832767 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:04.834456 (+ 1689us) service_pool.cc:227] Handling call
0807 03:39:04.890255 (+ 55799us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:04.954873 (+ 64618us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:04.954918 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:39:04.955370 (+ 452us) log.cc:842] Serialized 26 byte log entry
0807 03:39:05.238924 (+283554us) raft_consensus.cc:1631] Marking committed up to 1
0807 03:39:05.244299 (+ 5375us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:05.249923 (+ 5624us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:06.934708 (+1684785us) spinlock_profiling.cc:243] Waited 3.21 ms on lock 0x7b44000232cc. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f1a9a95 00007fe98f1aa4b2 00007fe991e52511 00007fe991e4de1a 00007fe991e41bee 00007fe991e40f36 00007fe991e40b58 00007fe991e4a551 00007fe991e4a509 00007fe991e4a499
0807 03:39:06.973539 (+ 38831us) raft_consensus.cc:1671] finished
0807 03:39:06.973561 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:06.973796 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6987,"raft.queue_time_us":7929,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":91,"spinlock_wait_cycles":7378688,"thread_start_us":277412,"threads_started":1,"wal-append.queue_time_us":287248}
I20260807 03:39:07.005172 18312 raft_consensus.cc:2454] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:39:07.155184 20399 raft_consensus.cc:3032] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:39:07.021687 20265 leader_election.cc:288] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:07.001940 20417 consensus_queue.cc:237] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:07.014168 19861 raft_consensus.cc:504] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:07.025177 20423 leader_election.cc:288] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:07.165407 20437 raft_consensus.cc:482] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:07.002717 20436 raft_consensus.cc:504] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:07.166492 20437 raft_consensus.cc:504] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:07.167743 19861 leader_election.cc:288] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:07.057979 20441 leader_election.cc:288] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:07.037716 20450 log.cc:924] Time spent T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 1.495s user 0.004s sys 0.005s
W20260807 03:39:07.039079 20458 log.cc:924] Time spent T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 1.496s user 0.004s sys 0.000s
W20260807 03:39:07.050565 20459 log.cc:924] Time spent T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 1.510s user 0.004s sys 0.000s
W20260807 03:39:07.044883 20474 log.cc:924] Time spent T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 1.166s user 0.000s sys 0.003s
W20260807 03:39:07.058531 20483 log.cc:924] Time spent T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.825s user 0.003s sys 0.000s
W20260807 03:39:07.057505 20489 log.cc:924] Time spent T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.754s user 0.002s sys 0.000s
I20260807 03:39:07.008061 20490 leader_election.cc:288] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:07.023033 20498 raft_consensus.cc:3037] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:07.016125 20500 raft_consensus.cc:482] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:07.323079 20500 raft_consensus.cc:3037] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:07.031212 20503 leader_election.cc:288] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:07.031735 20541 raft_consensus.cc:504] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:06.946172 20542 raft_consensus.cc:504] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:07.024219 20547 leader_election.cc:288] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:07.376605 18300 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57268 (request call id 946) took 1405 ms. Trace:
I20260807 03:39:06.991247 20551 raft_consensus.cc:504] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:07.377107 18300 rpcz_store.cc:276] 0807 03:39:05.971171 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:05.971838 (+ 667us) service_pool.cc:227] Handling call
0807 03:39:05.972259 (+ 421us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:05.980909 (+ 8650us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:05.980957 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:39:05.981525 (+ 568us) log.cc:842] Serialized 26 byte log entry
0807 03:39:06.079344 (+ 97819us) raft_consensus.cc:1631] Marking committed up to 1
0807 03:39:06.079384 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:06.079450 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:07.052257 (+972807us) spinlock_profiling.cc:243] Waited 11.9 ms on lock 0x7b44000232cc. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f1a9a95 00007fe98f1aa4b2 00007fe991e52511 00007fe991e4de1a 00007fe991e41bee 00007fe991e40f36 00007fe991e40b58 00007fe991e4a551 00007fe991e4a509 00007fe991e4a499
0807 03:39:07.374228 (+321971us) raft_consensus.cc:1671] finished
0807 03:39:07.374281 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:07.376405 (+ 2124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8134,"raft.queue_time_us":9063,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":27612,"spinlock_wait_cycles":27409152,"thread_start_us":55971,"threads_started":1,"wal-append.queue_time_us":230601}
I20260807 03:39:06.977233 20552 raft_consensus.cc:482] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:07.098672 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:07.143070 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:07.155797 20399 raft_consensus.cc:729] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:07.147334 20537 raft_consensus.cc:504] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:07.155879 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:07.168561 20436 leader_election.cc:288] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:07.010604 20205 raft_consensus.cc:504] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:07.181743 20437 leader_election.cc:288] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:07.219060 20042 leader_election.cc:288] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:07.229610 20543 raft_consensus.cc:482] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:07.219112 20252 raft_consensus.cc:482] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
W20260807 03:39:07.261618 18295 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57276 (request call id 697) took 1631 ms (client timeout 1728 ms). Trace:
W20260807 03:39:07.262830 18297 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57268 (request call id 928) took 2835 ms (client timeout 1980 ms). Trace:
I20260807 03:39:07.287542 18191 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44662 (request call id 671) took 1629 ms. Trace:
I20260807 03:39:07.304643 20433 raft_consensus.cc:482] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 03:39:07.305145 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 390313us
I20260807 03:39:07.305207 19874 leader_election.cc:288] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:07.305699 18310 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.501s user 0.022s sys 0.072s
W20260807 03:39:07.306155 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 443267us
I20260807 03:39:07.307046 20048 raft_consensus.cc:3037] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
W20260807 03:39:07.229038 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 46791us
I20260807 03:39:07.313081 18075 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33268 (request call id 918) took 1567 ms. Trace:
I20260807 03:39:07.330469 20144 raft_consensus.cc:482] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 03:39:07.338514 18006 thread.cc:636] client-negotiator [worker] (thread pool) Time spent creating pthread: real 1.955s user 0.006s sys 0.026s
I20260807 03:39:07.350543 19240 leader_election.cc:302] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331; no voters:
I20260807 03:39:07.357528 19292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:07.366840 20559 tablet_replica.cc:330] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02: stopping tablet replica
I20260807 03:39:07.369634 19295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:07.371316 18141 leader_election.cc:302] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:07.372437 20350 leader_election.cc:288] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:07.373871 18080 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33254 (request call id 1038) took 1723 ms. Trace:
I20260807 03:39:07.381736 20551 leader_election.cc:288] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:07.382251 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:07.387841 20076 raft_consensus.cc:482] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 03:39:07.404768 18295 rpcz_store.cc:269] 0807 03:39:05.629638 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:05.629765 (+ 127us) service_pool.cc:227] Handling call
0807 03:39:05.675733 (+ 45968us) spinlock_profiling.cc:243] Waited 3.16 ms on lock 0x7b58000e0498. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f25f851 00007fe98f25f702 00007fe99814fb09 00007fe99708b730 00007fe99715179b 00007fe997151821 00007fe9971011b0 00007fe997100a94 00007fe9907e8c04 00007fe9907e8b92
0807 03:39:07.261057 (+1585324us) spinlock_profiling.cc:243] Waited 975 ms on lock 0x7b54003e5654. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:07.261381 (+ 324us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2250871552}
I20260807 03:39:07.414515 20205 leader_election.cc:288] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:07.415268 20252 raft_consensus.cc:504] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:07.405309 18297 rpcz_store.cc:269] 0807 03:39:04.427585 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:04.427723 (+ 138us) service_pool.cc:227] Handling call
0807 03:39:07.262409 (+2834686us) spinlock_profiling.cc:243] Waited 288 ms on lock 0x7b54003e5154. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:07.262625 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":662955904}
I20260807 03:39:07.421510 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ca0223f47b46fd955704eba5259370" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:07.425519 19295 raft_consensus.cc:1799] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [term 3 FOLLOWER]: Handling vote request from an unknown peer 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:07.425617 18082 raft_consensus.cc:2454] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
W20260807 03:39:07.425930 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 55045.6us
I20260807 03:39:07.426240 19292 raft_consensus.cc:1799] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Handling vote request from an unknown peer 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:07.426625 20123 raft_consensus.cc:2781] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 03:39:07.442571 18082 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33268 (request call id 928) took 1049 ms. Trace:
I20260807 03:39:07.443022 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:07.442996 18082 rpcz_store.cc:276] 0807 03:39:06.390135 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:06.390242 (+ 107us) service_pool.cc:227] Handling call
0807 03:39:07.142547 (+752305us) spinlock_profiling.cc:243] Waited 3.05 ms on lock 0x7b5800001298. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f25f851 00007fe98f25f702 00007fe99814fb09 00007fe99708b730 00007fe99715179b 00007fe997151821 00007fe9971011b0 00007fe997100a94 00007fe9907e8c04 00007fe9907e8b92
0807 03:39:07.439808 (+297261us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7006592}
I20260807 03:39:07.443553 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:07.445235 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:07.445779 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:07.446678 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:07.447316 20258 raft_consensus.cc:482] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:07.422981 20437 raft_consensus.cc:482] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 03:39:07.427822 19240 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 87245us
I20260807 03:39:07.409869 18075 rpcz_store.cc:276] 0807 03:39:05.641813 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:05.642044 (+ 231us) service_pool.cc:227] Handling call
0807 03:39:06.108233 (+466189us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:06.128192 (+ 19959us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:07.156033 (+1027841us) spinlock_profiling.cc:243] Waited 466 ms on lock 0x7b540012d4d0. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bea09 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:07.209123 (+ 53090us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1071251840}
W20260807 03:39:07.418395 18006 thread.cc:602] client-negotiator [worker] (thread pool) Time spent starting thread: real 2.034s user 0.007s sys 0.026s
I20260807 03:39:07.422765 18080 rpcz_store.cc:276] 0807 03:39:05.623985 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:05.624172 (+ 187us) service_pool.cc:227] Handling call
0807 03:39:05.624552 (+ 380us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:05.624935 (+ 383us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:05.624977 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:39:05.625480 (+ 503us) log.cc:842] Serialized 26 byte log entry
0807 03:39:05.648780 (+ 23300us) raft_consensus.cc:1631] Marking committed up to 1
0807 03:39:05.648813 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:05.648871 (+ 58us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:07.043950 (+1395079us) spinlock_profiling.cc:243] Waited 3.16 ms on lock 0x7b44000232cc. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f1a9a95 00007fe98f1aa4b2 00007fe991e52511 00007fe991e4de1a 00007fe991e41bee 00007fe991e40f36 00007fe991e40b58 00007fe991e4a551 00007fe991e4a509 00007fe991e4a499
0807 03:39:07.309783 (+265833us) raft_consensus.cc:1671] finished
0807 03:39:07.312263 (+ 2480us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:07.347239 (+ 34976us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":792,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":3231,"spinlock_wait_cycles":7269632,"thread_start_us":13791,"threads_started":1,"wal-append.queue_time_us":14121}
I20260807 03:39:07.405421 18191 rpcz_store.cc:276] 0807 03:39:05.652428 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:05.656844 (+ 4416us) service_pool.cc:227] Handling call
0807 03:39:05.657133 (+ 289us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:05.916137 (+259004us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:05.916182 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:39:05.916676 (+ 494us) log.cc:842] Serialized 26 byte log entry
0807 03:39:06.045673 (+128997us) raft_consensus.cc:1631] Marking committed up to 2
0807 03:39:06.045816 (+ 143us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:06.859175 (+813359us) spinlock_profiling.cc:243] Waited 242 ms on lock 0x7b54001de554. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:39:06.859237 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:07.033395 (+174158us) spinlock_profiling.cc:243] Waited 15.5 ms on lock 0x7b44000232cc. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f1a9a95 00007fe98f1aa4b2 00007fe991e52511 00007fe991e4de1a 00007fe991e41bee 00007fe991e40f36 00007fe991e40b58 00007fe991e4a551 00007fe991e4a509 00007fe991e4a499
0807 03:39:07.278785 (+245390us) raft_consensus.cc:1671] finished
0807 03:39:07.278843 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:07.281306 (+ 2463us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16874,"raft.queue_time_us":26173,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":11113,"spinlock_wait_cycles":592277120,"thread_start_us":128704,"threads_started":1,"wal-append.queue_time_us":246393}
I20260807 03:39:07.432543 20076 raft_consensus.cc:504] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:07.433035 20517 raft_consensus.cc:2781] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:39:07.433584 20547 raft_consensus.cc:482] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:07.391960 20552 raft_consensus.cc:504] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:07.434521 20137 raft_consensus.cc:3037] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:07.434271 20433 raft_consensus.cc:504] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:07.438705 20252 leader_election.cc:288] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:07.438994 18188 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:50044 (request call id 1) took 2488 ms. Trace:
I20260807 03:39:07.426474 20144 raft_consensus.cc:504] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:07.448297 18300 raft_consensus.cc:2379] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 7: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:07.448868 20123 raft_consensus.cc:686] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:07.450184 20258 raft_consensus.cc:504] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:07.450996 20541 leader_election.cc:288] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:07.451090 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:07.451447 19295 raft_consensus.cc:2354] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3.
I20260807 03:39:07.417511 20399 consensus_queue.cc:260] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:07.417268 20543 raft_consensus.cc:504] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:07.458806 19292 raft_consensus.cc:2396] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260807 03:39:07.464038 18141 leader_election.cc:302] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:07.469137 20433 leader_election.cc:288] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:07.462877 20517 raft_consensus.cc:482] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:07.506654 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:07.506817 20517 raft_consensus.cc:3037] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:07.469508 20173 raft_consensus.cc:3032] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Stepping down as leader of term 5
I20260807 03:39:07.463383 18188 rpcz_store.cc:276] 0807 03:39:04.950271 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:04.959291 (+ 9020us) service_pool.cc:227] Handling call
0807 03:39:07.438435 (+2479144us) spinlock_profiling.cc:243] Waited 588 ms on lock 0x7b54002bedd4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c57c5 00007fe997101f39 00007fe9907ed704 00007fe9907ed692 00007fe9907ed5c1 00007fe9907ed53c 00007fe9907ec7f2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:07.438740 (+ 305us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1352315648}
I20260807 03:39:07.476511 20437 raft_consensus.cc:504] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:07.478257 20547 raft_consensus.cc:504] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:07.481448 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:07.484457 19496 raft_consensus.cc:482] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:07.484726 20123 consensus_queue.cc:237] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:07.486162 18138 leader_election.cc:398] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381) with higher term. Message: Invalid argument: T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3.
I20260807 03:39:07.487125 20076 leader_election.cc:288] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:07.516238 18138 leader_election.cc:401] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260807 03:39:07.487830 20144 leader_election.cc:288] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:07.489203 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:07.491618 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:07.499663 20111 leader_election.cc:288] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
W20260807 03:39:07.520531 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 72413.6us
I20260807 03:39:07.500138 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:07.501106 18187 raft_consensus.cc:1226] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 1. Current term is 2. Ops: []
I20260807 03:39:07.502067 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:07.502595 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:07.507412 18080 raft_consensus.cc:2454] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
W20260807 03:39:07.478829 20389 consensus_meta.cc:287] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 0.558s user 0.012s sys 0.003s
I20260807 03:39:07.514801 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:07.508411 18072 consensus_meta.cc:287] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 0.546s user 0.013s sys 0.003s
I20260807 03:39:07.523551 20111 raft_consensus.cc:2781] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:07.524384 18075 raft_consensus.cc:3037] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:07.524235 19496 raft_consensus.cc:504] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:07.529378 18071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33254 (request call id 1041) took 1667 ms. Trace:
I20260807 03:39:07.511472 20173 raft_consensus.cc:729] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:07.490821 20258 leader_election.cc:288] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:07.530053 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:07.530283 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:07.530810 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 102524us
I20260807 03:39:07.533035 20495 raft_consensus.cc:3037] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
W20260807 03:39:07.533999 18138 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.132:41381, user_credentials={real_user=slave}} blocked reactor thread for 48208.1us
I20260807 03:39:07.534691 20397 raft_consensus.cc:3037] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 3
I20260807 03:39:07.533179 20389 raft_consensus.cc:504] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:07.536528 20504 raft_consensus.cc:504] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:07.537729 20156 raft_consensus.cc:3037] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:07.539655 20367 raft_consensus.cc:3037] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:07.539083 20154 consensus_queue.cc:260] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:07.540902 18079 raft_consensus.cc:1226] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 4. Current term is 5. Ops: []
I20260807 03:39:07.541718 20437 leader_election.cc:288] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:07.541824 20576 log.cc:924] Time spent T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20260807 03:39:07.542129 20423 raft_consensus.cc:482] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:07.543339 20547 leader_election.cc:288] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:07.544946 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:07.504451 20542 leader_election.cc:288] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:07.553599 20111 raft_consensus.cc:482] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:07.555090 19659 raft_consensus.cc:3037] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:07.556622 19496 leader_election.cc:288] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:07.564067 18080 raft_consensus.cc:3037] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:07.565492 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:07.569944 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:07.598403 18081 raft_consensus.cc:2454] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
I20260807 03:39:07.574926 20173 consensus_queue.cc:260] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:07.553844 18071 rpcz_store.cc:276] 0807 03:39:05.861844 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:05.865396 (+ 3552us) service_pool.cc:227] Handling call
0807 03:39:05.905911 (+ 40515us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:06.173209 (+267298us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:06.173252 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:39:06.173716 (+ 464us) log.cc:842] Serialized 26 byte log entry
0807 03:39:06.300072 (+126356us) raft_consensus.cc:1631] Marking committed up to 3
0807 03:39:06.571163 (+271091us) log.cc:842] Serialized 14 byte log entry
0807 03:39:06.741313 (+170150us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:07.527283 (+785970us) spinlock_profiling.cc:243] Waited 249 ms on lock 0x7b5400034154. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:39:07.527384 (+ 101us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:07.527448 (+ 64us) raft_consensus.cc:1671] finished
0807 03:39:07.527563 (+ 115us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:07.529107 (+ 1544us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22765,"prepare.queue_time_us":374828,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":97535,"raft.run_cpu_time_us":271,"raft.run_wall_time_us":34821,"spinlock_wait_cycles":573562240,"thread_start_us":525671,"threads_started":3,"wal-append.queue_time_us":191331}
I20260807 03:39:07.581030 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:07.583223 18003 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica 3c4bceb836db4536a0f255fdf86abc02 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:07.602052 18309 raft_consensus.cc:2379] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 4: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:07.589560 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:07.512420 20583 raft_consensus.cc:482] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:07.573724 18193 raft_consensus.cc:2932] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:07.603273 19297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:07.601151 20423 raft_consensus.cc:504] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:07.588397 20399 raft_consensus.cc:3037] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:07.612787 18311 raft_consensus.cc:1261] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:39:07.601583 18082 raft_consensus.cc:2454] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:39:07.569535 18072 raft_consensus.cc:2454] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 3.
I20260807 03:39:07.604643 20583 raft_consensus.cc:504] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:07.608044 20570 log.cc:924] Time spent T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.147s user 0.000s sys 0.001s
I20260807 03:39:07.607807 19295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:07.608192 18187 raft_consensus.cc:3037] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:07.608445 20136 raft_consensus.cc:482] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:07.609012 20423 leader_election.cc:288] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:07.591830 18297 raft_consensus.cc:2454] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:39:07.615438 18080 raft_consensus.cc:2454] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 7.
I20260807 03:39:07.618156 20509 raft_consensus.cc:482] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:07.618151 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:07.624226 20564 raft_consensus.cc:482] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:07.601557 20111 raft_consensus.cc:3037] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:07.621590 18300 raft_consensus.cc:2454] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3.
I20260807 03:39:07.601281 20258 raft_consensus.cc:482] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:07.631278 20136 raft_consensus.cc:504] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:07.631902 18140 leader_election.cc:302] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:07.633584 18140 leader_election.cc:302] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:07.638866 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:07.619728 20154 raft_consensus.cc:3037] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:07.638810 20365 raft_consensus.cc:2726] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:07.643101 20124 raft_consensus.cc:2781] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:39:07.643360 18309 raft_consensus.cc:2454] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:39:07.643271 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:07.626876 20076 raft_consensus.cc:482] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 03:39:07.645004 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 52415us
I20260807 03:39:07.647203 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:07.650456 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:07.650626 20356 raft_consensus.cc:482] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:07.651149 18297 raft_consensus.cc:2454] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:39:07.605621 20543 leader_election.cc:288] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:07.652029 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:07.640385 18027 leader_election.cc:398] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Vote denied by peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021) with higher term. Message: Invalid argument: T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 2. Current term is 3.
I20260807 03:39:07.657222 18027 leader_election.cc:401] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260807 03:39:07.562513 20537 leader_election.cc:288] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:07.633610 20136 leader_election.cc:288] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:07.628400 20552 leader_election.cc:288] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:07.663477 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:07.664976 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:07.652278 20076 raft_consensus.cc:504] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:07.665829 20558 raft_consensus.cc:482] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:07.665767 20357 consensus_queue.cc:1046] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20260807 03:39:07.652765 20173 raft_consensus.cc:3037] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:07.651530 20356 raft_consensus.cc:504] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:07.648864 20124 raft_consensus.cc:482] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:07.684379 20357 raft_consensus.cc:3032] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 03:39:07.684782 20357 raft_consensus.cc:729] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:07.652303 20233 raft_consensus.cc:482] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:07.690027 20073 raft_consensus.cc:482] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:07.670456 20558 raft_consensus.cc:504] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:07.701772 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:07.702656 18187 raft_consensus.cc:2396] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260807 03:39:07.704370 18311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57268 (request call id 930) took 3133 ms. Trace:
I20260807 03:39:07.683431 20124 raft_consensus.cc:3037] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:07.706051 20530 consensus_queue.cc:1035] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20260807 03:39:07.710707 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:07.712203 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:07.707059 18311 rpcz_store.cc:276] 0807 03:39:04.564296 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:04.590954 (+ 26658us) service_pool.cc:227] Handling call
0807 03:39:04.643286 (+ 52332us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:07.619251 (+2975965us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:07.677203 (+ 57952us) spinlock_profiling.cc:243] Waited 2.97 s on lock 0x7b54002b4dd4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:39:07.698040 (+ 20837us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6827778816}
I20260807 03:39:07.650630 19239 leader_election.cc:302] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:07.718849 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:07.642117 20564 raft_consensus.cc:504] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:07.727905 18180 raft_consensus.cc:2379] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 7: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:07.724035 18075 raft_consensus.cc:2454] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:39:07.676662 20258 raft_consensus.cc:504] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:07.734014 20558 leader_election.cc:288] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:07.735937 20494 raft_consensus.cc:482] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:07.739089 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:07.634163 20509 raft_consensus.cc:504] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:07.741977 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:07.740020 18189 raft_consensus.cc:2454] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
I20260807 03:39:07.739228 20494 raft_consensus.cc:504] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:07.748059 19617 raft_consensus.cc:482] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:07.748733 19617 raft_consensus.cc:504] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:07.752439 18305 raft_consensus.cc:2454] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:39:07.755028 18305 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57268 (request call id 954) took 1190 ms. Trace:
I20260807 03:39:07.756533 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:07.757146 20076 leader_election.cc:288] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:07.758034 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:07.759434 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:07.765857 18311 raft_consensus.cc:2379] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 4: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:07.767040 18297 raft_consensus.cc:2379] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 5: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:07.714454 20389 leader_election.cc:288] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:07.772634 20550 log.cc:924] Time spent T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.223s user 0.000s sys 0.002s
I20260807 03:39:07.776438 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:07.777195 18309 raft_consensus.cc:2454] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:39:07.777706 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:07.755429 18305 rpcz_store.cc:276] 0807 03:39:06.564155 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:06.564713 (+ 558us) service_pool.cc:227] Handling call
0807 03:39:07.754795 (+1190082us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b5400031bd4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:07.754967 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2670689408}
I20260807 03:39:07.737352 20504 leader_election.cc:288] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:07.783133 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:07.798676 20509 leader_election.cc:288] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:07.799921 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:07.801000 18312 raft_consensus.cc:3037] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:07.802809 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:07.803385 18297 raft_consensus.cc:2454] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 2.
I20260807 03:39:07.799715 20258 leader_election.cc:288] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:07.783044 20277 raft_consensus.cc:3037] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:07.736728 20073 raft_consensus.cc:504] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:07.781816 19617 leader_election.cc:288] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:07.729746 20233 raft_consensus.cc:504] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:07.776075 20500 raft_consensus.cc:504] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:07.840562 18248 leader_election.cc:302] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:07.855182 18312 raft_consensus.cc:2454] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 6.
I20260807 03:39:07.878485 18180 raft_consensus.cc:2454] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
W20260807 03:39:07.809894 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 207652us
I20260807 03:39:07.837651 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:07.813480 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:07.807677 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:07.840041 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:07.906314 18309 raft_consensus.cc:2454] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 2.
I20260807 03:39:07.913758 18309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57276 (request call id 708) took 1001 ms. Trace:
I20260807 03:39:07.775463 20496 raft_consensus.cc:2726] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021) with higher term. Message: Invalid argument: T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 2. Current term is 3.
I20260807 03:39:07.788486 20586 raft_consensus.cc:482] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:07.914965 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:39:07.876001 20347 consensus_meta.cc:287] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 0.707s user 0.012s sys 0.006s
I20260807 03:39:07.910853 20173 raft_consensus.cc:482] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:07.917313 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:07.868782 20564 leader_election.cc:288] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:07.925696 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:07.923791 18309 rpcz_store.cc:276] 0807 03:39:06.912816 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:07.784230 (+871414us) service_pool.cc:227] Handling call
0807 03:39:07.913689 (+129459us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 03:39:07.932392 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:07.936256 18141 leader_election.cc:302] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
W20260807 03:39:07.942162 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 89190.2us
I20260807 03:39:07.826357 20583 leader_election.cc:288] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:07.947971 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:07.848516 20076 raft_consensus.cc:482] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:07.924793 18305 raft_consensus.cc:2454] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 2.
I20260807 03:39:07.967671 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:07.927897 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 122638us
I20260807 03:39:07.826452 20356 leader_election.cc:288] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:07.969547 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:07.976619 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:07.976768 20556 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:07.977169 20205 raft_consensus.cc:482] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:07.977519 20173 raft_consensus.cc:504] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:07.978482 18305 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57276 (request call id 710) took 1065 ms. Trace:
I20260807 03:39:07.986418 18305 rpcz_store.cc:276] 0807 03:39:06.913419 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:07.783536 (+870117us) service_pool.cc:227] Handling call
0807 03:39:07.978226 (+194690us) spinlock_profiling.cc:243] Waited 13.5 ms on lock 0x7b540018bad4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:07.978402 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":30977408}
I20260807 03:39:07.978780 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:07.979220 20496 raft_consensus.cc:482] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:07.966116 18310 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57268 (request call id 960) took 1393 ms. Trace:
I20260807 03:39:07.997262 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:39:07.998726 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 72291.7us
I20260807 03:39:07.994697 18300 raft_consensus.cc:2396] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 03:39:07.996977 20205 raft_consensus.cc:504] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:07.926615 20537 raft_consensus.cc:482] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:08.002652 18189 raft_consensus.cc:2454] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:39:08.009768 18248 leader_election.cc:302] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:08.015893 18300 raft_consensus.cc:1226] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:08.019490 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:08.016256 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:08.020344 18188 raft_consensus.cc:2454] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
I20260807 03:39:08.019814 17912 catalog_manager.cc:5495] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:08.021252 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:08.021479 17912 catalog_manager.cc:5495] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 1 to 3, leader changed from d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 3 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:39:08.022449 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:08.023054 18300 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57268 (request call id 964) took 1154 ms. Trace:
I20260807 03:39:08.020690 18309 raft_consensus.cc:3037] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:08.024355 18297 raft_consensus.cc:2454] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3.
I20260807 03:39:08.026554 20233 leader_election.cc:288] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:08.023643 18300 rpcz_store.cc:276] 0807 03:39:06.868684 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:08.015218 (+1146534us) service_pool.cc:227] Handling call
0807 03:39:08.015636 (+ 418us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:08.019712 (+ 4076us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:08.022951 (+ 3239us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 03:39:08.014364 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:08.002421 20496 raft_consensus.cc:504] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:08.028982 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:07.997756 18310 rpcz_store.cc:276] 0807 03:39:06.572859 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:06.598225 (+ 25366us) service_pool.cc:227] Handling call
0807 03:39:06.598680 (+ 455us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 03:39:06.768525 (+169845us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:06.768561 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 03:39:06.769202 (+ 641us) log.cc:842] Serialized 52 byte log entry
0807 03:39:07.426863 (+657661us) raft_consensus.cc:1631] Marking committed up to 1
0807 03:39:07.426985 (+ 122us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:07.427099 (+ 114us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:07.950455 (+523356us) raft_consensus.cc:1671] finished
0807 03:39:07.950475 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:07.966050 (+ 15575us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":36684,"raft.queue_time_us":92154,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":112,"thread_start_us":765004,"threads_started":2,"wal-append.queue_time_us":590861}
I20260807 03:39:08.032732 18185 raft_consensus.cc:1261] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260807 03:39:08.034035 18305 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57276 (request call id 705) took 1134 ms. Trace:
I20260807 03:39:08.034490 18305 rpcz_store.cc:276] 0807 03:39:06.899104 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:08.032890 (+1133786us) service_pool.cc:227] Handling call
0807 03:39:08.033298 (+ 408us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:08.033638 (+ 340us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:08.033674 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:08.033724 (+ 50us) raft_consensus.cc:1631] Marking committed up to 2
0807 03:39:08.033766 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:08.033828 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:08.033954 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 03:39:08.035295 18298 raft_consensus.cc:2454] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:39:08.036640 18298 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57268 (request call id 953) took 1472 ms. Trace:
I20260807 03:39:08.036959 18298 rpcz_store.cc:276] 0807 03:39:06.563775 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:06.564639 (+ 864us) service_pool.cc:227] Handling call
0807 03:39:08.036463 (+1471824us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b54001e64d4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:08.036593 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3182977920}
I20260807 03:39:08.037535 20496 leader_election.cc:288] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:08.038076 20076 raft_consensus.cc:504] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:08.024312 18191 raft_consensus.cc:2454] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:39:08.038419 18298 raft_consensus.cc:1261] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260807 03:39:08.043141 20076 leader_election.cc:288] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:08.056886 18298 pending_rounds.cc:76] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49: Aborting all ops after (but not including) 2
I20260807 03:39:08.043574 20448 raft_consensus.cc:482] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:08.046610 18140 leader_election.cc:302] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:08.045771 20205 leader_election.cc:288] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:08.048759 20173 leader_election.cc:288] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:08.052533 20494 leader_election.cc:288] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:08.057466 18298 pending_rounds.cc:97] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49: Aborting uncommitted NO_OP operation due to leader change: 3.3
W20260807 03:39:08.055373 20281 consensus_meta.cc:287] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.665s user 0.013s sys 0.005s
I20260807 03:39:08.066704 18298 raft_consensus.cc:2866] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 03:39:08.054243 20124 raft_consensus.cc:504] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:08.067036 20281 raft_consensus.cc:504] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:08.067751 18184 raft_consensus.cc:2454] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 6.
I20260807 03:39:08.069041 20281 leader_election.cc:288] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:08.069239 18298 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57276 (request call id 707) took 1162 ms. Trace:
I20260807 03:39:08.042088 18297 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57268 (request call id 965) took 1173 ms. Trace:
I20260807 03:39:08.069959 18298 rpcz_store.cc:276] 0807 03:39:06.907149 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:08.037701 (+1130552us) service_pool.cc:227] Handling call
0807 03:39:08.038074 (+ 373us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:08.067483 (+ 29409us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:08.069193 (+ 1710us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 03:39:08.070282 18297 rpcz_store.cc:276] 0807 03:39:06.868973 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:08.025060 (+1156087us) service_pool.cc:227] Handling call
0807 03:39:08.025404 (+ 344us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:08.040845 (+ 15441us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:08.040898 (+ 53us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:08.040961 (+ 63us) raft_consensus.cc:1631] Marking committed up to 2
0807 03:39:08.041030 (+ 69us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:08.041770 (+ 740us) spinlock_profiling.cc:243] Waited 15.2 ms on lock 0x7b54003e6054. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:39:08.041828 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:08.041991 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":34852096}
I20260807 03:39:08.072239 18298 raft_consensus.cc:1226] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 5. Current term is 6. Ops: [5.4-5.4]
W20260807 03:39:08.072742 20517 consensus_meta.cc:287] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331: Time spent flushing consensus metadata: real 0.562s user 0.003s sys 0.010s
I20260807 03:39:08.072901 18298 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57276 (request call id 713) took 1158 ms. Trace:
I20260807 03:39:08.072966 18297 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57276 (request call id 718) took 1085 ms. Trace:
W20260807 03:39:08.061041 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 44725.1us
I20260807 03:39:08.038767 18180 raft_consensus.cc:2454] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:39:08.052438 18309 raft_consensus.cc:2454] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 5.
I20260807 03:39:08.042675 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:08.070021 20537 raft_consensus.cc:504] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:08.055264 18310 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57276 (request call id 711) took 1141 ms. Trace:
I20260807 03:39:08.038064 18305 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57276 (request call id 706) took 1138 ms. Trace:
I20260807 03:39:08.069788 18295 raft_consensus.cc:2354] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 6. Current term is 7.
I20260807 03:39:08.000039 19617 raft_consensus.cc:2781] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:39:08.080713 18305 rpcz_store.cc:276] 0807 03:39:06.899345 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:08.035750 (+1136405us) service_pool.cc:227] Handling call
0807 03:39:08.036883 (+ 1133us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:08.037269 (+ 386us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:08.037330 (+ 61us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:08.037402 (+ 72us) raft_consensus.cc:1631] Marking committed up to 1
0807 03:39:08.037524 (+ 122us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:08.037640 (+ 116us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:08.037888 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 03:39:07.842810 20357 consensus_queue.cc:260] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:08.042462 20389 raft_consensus.cc:2781] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:39:08.057917 20448 raft_consensus.cc:504] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:08.065864 20552 raft_consensus.cc:2781] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:39:08.073323 18298 rpcz_store.cc:276] 0807 03:39:06.914303 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:08.071389 (+1157086us) service_pool.cc:227] Handling call
0807 03:39:08.071935 (+ 546us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:08.072673 (+ 738us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:08.072850 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 03:39:08.074143 18297 rpcz_store.cc:276] 0807 03:39:06.987103 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:08.071804 (+1084701us) service_pool.cc:227] Handling call
0807 03:39:08.072259 (+ 455us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:08.072595 (+ 336us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:08.072629 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:08.072679 (+ 50us) raft_consensus.cc:1631] Marking committed up to 2
0807 03:39:08.072715 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:08.072778 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:08.072905 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 03:39:08.049252 18300 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57268 (request call id 970) took 1157 ms. Trace:
I20260807 03:39:08.023334 17912 catalog_manager.cc:5495] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 1 to 3, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 3 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:08.080513 18310 rpcz_store.cc:276] 0807 03:39:06.913690 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:08.054486 (+1140796us) service_pool.cc:227] Handling call
0807 03:39:08.054796 (+ 310us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:08.054967 (+ 171us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:08.054993 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:08.055032 (+ 39us) raft_consensus.cc:1631] Marking committed up to 2
0807 03:39:08.055062 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:08.055110 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:08.055198 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 03:39:08.108285 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:08.077358 18188 raft_consensus.cc:2454] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
I20260807 03:39:08.087116 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:08.089042 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:08.050513 18312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57268 (request call id 971) took 1152 ms. Trace:
I20260807 03:39:08.111876 18300 rpcz_store.cc:276] 0807 03:39:06.891718 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:08.029148 (+1137430us) service_pool.cc:227] Handling call
0807 03:39:08.048519 (+ 19371us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:08.048808 (+ 289us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:08.048846 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:08.048910 (+ 64us) raft_consensus.cc:1631] Marking committed up to 2
0807 03:39:08.048956 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:08.049032 (+ 76us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:08.049174 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 03:39:08.113246 18309 raft_consensus.cc:1226] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 4. Current term is 5. Ops: []
I20260807 03:39:08.075169 18189 raft_consensus.cc:2454] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
I20260807 03:39:07.809303 20498 raft_consensus.cc:504] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:08.114416 18295 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57276 (request call id 712) took 1200 ms. Trace:
I20260807 03:39:08.094744 18311 raft_consensus.cc:1799] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Handling vote request from an unknown peer 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:08.129786 18295 rpcz_store.cc:276] 0807 03:39:06.914030 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:07.445407 (+531377us) service_pool.cc:227] Handling call
0807 03:39:08.114196 (+668789us) spinlock_profiling.cc:243] Waited 623 ms on lock 0x7b54001d1354. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:08.114338 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1433194880}
I20260807 03:39:08.112984 18312 rpcz_store.cc:276] 0807 03:39:06.898220 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:08.030509 (+1132289us) service_pool.cc:227] Handling call
0807 03:39:08.049742 (+ 19233us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:08.050046 (+ 304us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:08.050085 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:08.050160 (+ 75us) raft_consensus.cc:1631] Marking committed up to 1
0807 03:39:08.050208 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:08.050313 (+ 105us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:08.050435 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 03:39:08.091980 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 101306us
I20260807 03:39:08.118258 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:08.118857 19704 raft_consensus.cc:482] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:08.119588 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:08.120774 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:08.122162 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:08.125563 19617 raft_consensus.cc:482] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:08.126629 18190 raft_consensus.cc:2454] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:08.069527 20124 leader_election.cc:288] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:08.128834 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:08.016957 20556 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:08.079872 20537 leader_election.cc:288] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:08.207211 18069 raft_consensus.cc:2379] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5c9b5062bdfa46af9396f1a85025a331 in current term 3: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:08.207018 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:08.075932 20397 raft_consensus.cc:2726] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381) with higher term. Message: Invalid argument: T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3.
I20260807 03:39:08.215914 18073 raft_consensus.cc:2454] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:39:08.217707 18309 raft_consensus.cc:1226] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 4. Current term is 5. Ops: []
I20260807 03:39:08.196199 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:08.143191 18311 raft_consensus.cc:2354] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3.
I20260807 03:39:08.220148 18300 raft_consensus.cc:1226] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3. Ops: []
I20260807 03:39:08.077443 20399 consensus_queue.cc:1035] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.095s
W20260807 03:39:08.143726 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 112573us
I20260807 03:39:08.225766 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:08.111984 20552 raft_consensus.cc:482] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 03:39:08.227785 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 245162us
I20260807 03:39:08.007954 20586 raft_consensus.cc:504] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:08.115491 20389 raft_consensus.cc:482] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:08.232237 18082 raft_consensus.cc:2454] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 5.
I20260807 03:39:08.234899 18072 raft_consensus.cc:2454] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:39:08.254690 18081 raft_consensus.cc:2379] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 2: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:08.195981 17912 catalog_manager.cc:5495] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 1 to 3, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 3 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:08.219803 19617 raft_consensus.cc:3037] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:08.241295 18075 raft_consensus.cc:2454] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:39:08.222447 19704 raft_consensus.cc:504] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:08.073118 20517 raft_consensus.cc:504] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:08.243321 18027 leader_election.cc:302] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:08.247882 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:08.215337 20073 leader_election.cc:288] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
W20260807 03:39:08.225939 20111 consensus_meta.cc:287] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 0.578s user 0.005s sys 0.005s
I20260807 03:39:08.249752 18141 leader_election.cc:302] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:08.230367 18080 raft_consensus.cc:3037] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:08.232491 20543 raft_consensus.cc:2726] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 03:39:08.252902 18297 raft_consensus.cc:1261] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260807 03:39:08.270008 18073 raft_consensus.cc:2379] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 2: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:08.248256 20357 raft_consensus.cc:3037] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:08.257460 20389 raft_consensus.cc:3037] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:08.281041 18141 leader_election.cc:302] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:08.262614 18079 raft_consensus.cc:2454] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
W20260807 03:39:08.283272 19471 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.630s user 0.025s sys 0.028s
I20260807 03:39:08.288285 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:08.268545 20500 leader_election.cc:288] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:08.269670 20397 raft_consensus.cc:482] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:08.263234 20552 raft_consensus.cc:3037] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
W20260807 03:39:08.283660 19471 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.630s user 0.025s sys 0.028s
I20260807 03:39:08.286327 20124 raft_consensus.cc:2781] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:39:08.272308 18183 raft_consensus.cc:2454] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
I20260807 03:39:08.287592 20537 raft_consensus.cc:2781] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:08.309772 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:08.310640 18079 raft_consensus.cc:2454] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:39:08.297953 19297 raft_consensus.cc:2379] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 3: Already voted for candidate 5c9b5062bdfa46af9396f1a85025a331 in this term.
I20260807 03:39:08.311368 20397 raft_consensus.cc:504] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:08.313069 20517 leader_election.cc:288] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:08.313653 20517 raft_consensus.cc:2781] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:39:08.313769 20397 leader_election.cc:288] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:08.314131 20517 raft_consensus.cc:686] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [term 3 LEADER]: Becoming Leader. State: Replica: 5c9b5062bdfa46af9396f1a85025a331, State: Running, Role: LEADER
I20260807 03:39:08.315100 20517 consensus_queue.cc:237] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:08.317387 18069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58950 (request call id 14) took 1221 ms. Trace:
I20260807 03:39:08.317729 18295 raft_consensus.cc:1261] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 03:39:08.317737 18069 rpcz_store.cc:276] 0807 03:39:07.096200 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:07.097540 (+ 1340us) service_pool.cc:227] Handling call
0807 03:39:08.317212 (+1219672us) spinlock_profiling.cc:243] Waited 814 ms on lock 0x7b54002c4cd4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:08.317334 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1871263232}
I20260807 03:39:08.319977 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:08.320772 18073 raft_consensus.cc:3037] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:08.299916 20367 raft_consensus.cc:2781] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:08.311769 20556 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:08.309583 18027 leader_election.cc:302] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:08.304646 20124 raft_consensus.cc:482] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:08.270687 20111 raft_consensus.cc:504] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:08.299084 17912 catalog_manager.cc:5495] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 1 to 3, leader changed from d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 3 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:39:08.299921 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:08.346342 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
W20260807 03:39:08.329838 19292 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.782s user 0.018s sys 0.059s
I20260807 03:39:08.363019 18027 leader_election.cc:302] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:08.364435 18187 raft_consensus.cc:2354] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 4. Current term is 5.
I20260807 03:39:08.326988 20367 raft_consensus.cc:482] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:08.364517 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:08.367686 18080 raft_consensus.cc:2454] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:39:08.368813 18248 leader_election.cc:302] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:08.334797 20124 raft_consensus.cc:3037] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:08.334066 20494 raft_consensus.cc:2726] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:08.331046 20498 leader_election.cc:288] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
W20260807 03:39:08.361902 20495 consensus_meta.cc:287] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 0.776s user 0.013s sys 0.003s
I20260807 03:39:08.389295 18141 leader_election.cc:302] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
W20260807 03:39:08.370234 19292 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.818s user 0.018s sys 0.059s
I20260807 03:39:08.387401 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ca0223f47b46fd955704eba5259370" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:08.377663 20173 raft_consensus.cc:482] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:08.420275 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:08.379285 20367 raft_consensus.cc:3037] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:08.366007 19704 leader_election.cc:288] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:08.397264 18027 leader_election.cc:302] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:08.397678 18248 leader_election.cc:302] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:08.428737 18188 raft_consensus.cc:3037] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
W20260807 03:39:08.428930 20503 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.754s user 0.027s sys 0.013s
W20260807 03:39:08.382954 19239 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.639s user 0.036s sys 0.036s
I20260807 03:39:08.442261 20173 raft_consensus.cc:504] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:08.442979 19239 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.699s user 0.036s sys 0.037s
W20260807 03:39:08.443578 19239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 793203us
I20260807 03:39:08.429593 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:08.444779 20173 leader_election.cc:288] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:08.440652 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:08.390460 17912 catalog_manager.cc:5495] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 2 to 3, leader changed from d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 3 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:08.382659 20496 raft_consensus.cc:2781] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:39:08.435817 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:08.437266 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 44429.3us
I20260807 03:39:08.424763 20397 raft_consensus.cc:2781] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260807 03:39:08.440052 20503 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.907s user 0.027s sys 0.013s
I20260807 03:39:08.451030 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:08.440625 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:08.428941 19295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331"
W20260807 03:39:08.430037 18003 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet b58a8dc36ec94e0c8877bdf0f901598a, TS d9716e2bf06442f5b3b0e81087c53c4d -> TS 5c9b5062bdfa46af9396f1a85025a331 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:08.454990 18069 raft_consensus.cc:2379] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5c9b5062bdfa46af9396f1a85025a331 in current term 3: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:08.441120 18293 raft_consensus.cc:2454] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
W20260807 03:39:08.441607 18249 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.575s user 0.010s sys 0.035s
W20260807 03:39:08.455719 18249 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.673s user 0.010s sys 0.035s
I20260807 03:39:08.442596 20281 raft_consensus.cc:2781] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:39:08.398401 20277 raft_consensus.cc:2781] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:08.461131 20277 raft_consensus.cc:482] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:08.461567 20277 raft_consensus.cc:3037] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:08.462492 20369 raft_consensus.cc:482] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:08.449184 18248 leader_election.cc:302] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
W20260807 03:39:08.455108 18003 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet b58a8dc36ec94e0c8877bdf0f901598a, TS d9716e2bf06442f5b3b0e81087c53c4d -> TS 5c9b5062bdfa46af9396f1a85025a331 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:08.378216 20556 consensus_queue.cc:1035] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.057s
I20260807 03:39:08.427659 20042 raft_consensus.cc:482] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:08.428020 18141 leader_election.cc:302] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:08.429603 20500 raft_consensus.cc:482] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:08.458122 18079 raft_consensus.cc:2454] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 5.
I20260807 03:39:08.450564 20496 raft_consensus.cc:482] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 03:39:08.456290 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 724311us
I20260807 03:39:08.431519 20494 raft_consensus.cc:482] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
W20260807 03:39:08.456951 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 76833.4us
I20260807 03:39:08.451256 20397 raft_consensus.cc:482] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:08.460387 20281 raft_consensus.cc:482] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:08.464737 20173 raft_consensus.cc:2781] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:39:08.487879 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:08.491117 18080 raft_consensus.cc:2454] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:39:08.489912 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:08.493271 18072 raft_consensus.cc:2454] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
W20260807 03:39:08.468130 18193 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.565s user 0.022s sys 0.027s
W20260807 03:39:08.495683 19743 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.020s user 0.018s sys 0.040s
I20260807 03:39:08.479763 20399 raft_consensus.cc:482] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 03:39:08.496062 19743 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.057s user 0.018s sys 0.040s
I20260807 03:39:08.496289 20399 raft_consensus.cc:504] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:08.497264 20156 raft_consensus.cc:2781] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 03:39:08.497738 20156 raft_consensus.cc:686] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:08.498337 20399 leader_election.cc:288] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:08.499214 18069 raft_consensus.cc:1226] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 1. Current term is 2. Ops: []
I20260807 03:39:08.488171 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:08.500164 18193 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.597s user 0.023s sys 0.027s
I20260807 03:39:08.489069 18188 raft_consensus.cc:2454] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 3.
I20260807 03:39:08.492177 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:08.458228 20586 leader_election.cc:288] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:08.502401 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:08.503165 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:08.492730 18027 leader_election.cc:302] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:08.504868 18079 raft_consensus.cc:1799] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Handling vote request from an unknown peer 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:08.506079 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:08.508538 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:08.510138 18079 raft_consensus.cc:2354] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3.
I20260807 03:39:08.510108 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:08.480937 20397 raft_consensus.cc:3037] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:08.473165 20496 raft_consensus.cc:3037] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:08.488664 19704 raft_consensus.cc:482] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:08.507287 17912 catalog_manager.cc:5495] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 1 to 3, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 3 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
W20260807 03:39:08.520213 20546 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.628s user 0.023s sys 0.019s
I20260807 03:39:08.517424 18248 leader_election.cc:302] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:08.536865 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:08.537665 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 109765us
I20260807 03:39:08.465231 20369 raft_consensus.cc:504] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:08.536829 18185 raft_consensus.cc:2379] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 6: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:08.492398 20281 raft_consensus.cc:3037] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:08.537897 18069 raft_consensus.cc:2379] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 3: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:08.492893 20173 raft_consensus.cc:482] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 03:39:08.537287 20546 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.710s user 0.023s sys 0.020s
I20260807 03:39:08.490254 20500 raft_consensus.cc:504] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:08.474871 20042 raft_consensus.cc:504] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:08.540349 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 59189.6us
I20260807 03:39:08.536924 19704 raft_consensus.cc:504] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:08.522434 19240 leader_election.cc:302] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:08.544833 18075 raft_consensus.cc:2396] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 03:39:08.568113 18027 leader_election.cc:302] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:08.563902 18082 raft_consensus.cc:2454] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:39:08.511461 20399 raft_consensus.cc:2781] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:08.502691 20154 raft_consensus.cc:482] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:08.552983 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:08.571121 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:39:08.549754 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 59700.5us
I20260807 03:39:08.555131 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:08.539461 18079 raft_consensus.cc:1261] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 03:39:08.564054 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:08.562786 18141 leader_election.cc:302] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:08.567137 18073 raft_consensus.cc:2454] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 6.
W20260807 03:39:08.547616 20123 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.994s user 0.044s sys 0.096s
I20260807 03:39:08.551216 20111 leader_election.cc:288] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:08.552678 20586 raft_consensus.cc:2781] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:39:08.581735 18188 raft_consensus.cc:2379] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 7: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:08.513386 20494 raft_consensus.cc:504] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:08.557582 20173 raft_consensus.cc:3037] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:08.586942 19704 leader_election.cc:288] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:08.586790 20154 raft_consensus.cc:504] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:08.588881 20575 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:39:05.301076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.198.130:37057 (local address 127.0.0.1:45378)
0807 03:39:07.138687 (+1837611us) spinlock_profiling.cc:243] Waited 15.6 ms on lock 0x7b44000232cc. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f1a9a95 00007fe98f1aa4b2 00007fe991e52511 00007fe991e4de1a 00007fe98f2795fe 00007fe9980b0a39 00007fe98f2968bb 00007fe98f294cbf 00007fe98f296e22 00007fe99058fc48
0807 03:39:07.679900 (+541213us) negotiation.cc:107] Waiting for socket to connect
0807 03:39:07.691495 (+ 11595us) client_negotiation.cc:174] Beginning negotiation
0807 03:39:07.858793 (+167298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:39:07.908103 (+ 49310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:39:07.911493 (+ 3390us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:39:07.966196 (+ 54703us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:39:08.002716 (+ 36520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:39:08.002823 (+ 107us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:39:08.370069 (+367246us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.15.198.130:37057: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":2206912,"spinlock_wait_cycles":35976576,"thread_start_us":2075869,"threads_started":1}
I20260807 03:39:08.615322 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:08.589919 20500 leader_election.cc:288] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:08.568444 20585 log.cc:924] Time spent T 00000000000000000000000000000000 P 8da2065a77024f35bdfc0dc3adb83f69: Append to log took a long time: real 0.311s user 0.000s sys 0.002s
I20260807 03:39:08.591799 20495 raft_consensus.cc:2781] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:08.576700 20399 raft_consensus.cc:482] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:08.592536 18189 raft_consensus.cc:2454] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
I20260807 03:39:08.620196 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:08.620635 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:08.620905 18072 raft_consensus.cc:2454] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
W20260807 03:39:08.594458 18003 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 28ccf467b7d24422ba092003f6b06616, TS 3c4bceb836db4536a0f255fdf86abc02 -> TS 5c9b5062bdfa46af9396f1a85025a331 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:08.621475 18003 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 28ccf467b7d24422ba092003f6b06616, TS 3c4bceb836db4536a0f255fdf86abc02 -> TS 5c9b5062bdfa46af9396f1a85025a331 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:08.562249 20543 raft_consensus.cc:2781] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:39:08.582186 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:08.605711 20586 raft_consensus.cc:482] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:08.606302 18185 raft_consensus.cc:2379] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 3: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:08.608718 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:08.581260 20123 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.028s user 0.044s sys 0.097s
I20260807 03:39:08.611554 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:08.638434 18079 raft_consensus.cc:2354] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 6. Current term is 7.
I20260807 03:39:08.612439 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:08.639187 18080 raft_consensus.cc:3037] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 7
I20260807 03:39:08.505508 20156 consensus_queue.cc:237] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:08.607012 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:08.624145 20495 raft_consensus.cc:482] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:08.626753 20154 leader_election.cc:288] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:08.615626 20111 raft_consensus.cc:2726] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:08.584684 18191 raft_consensus.cc:2454] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
I20260807 03:39:08.630579 17912 catalog_manager.cc:5495] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 1 to 2, leader changed from d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 2 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:08.642964 18081 raft_consensus.cc:1799] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Handling vote request from an unknown peer 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:08.653012 18081 raft_consensus.cc:2396] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260807 03:39:08.644507 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:39:08.627312 18310 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.502s user 0.050s sys 0.032s
W20260807 03:39:08.656968 18310 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.532s user 0.051s sys 0.033s
I20260807 03:39:08.623950 20399 raft_consensus.cc:3037] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:08.651786 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:08.653586 18189 raft_consensus.cc:2454] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 6.
I20260807 03:39:08.639642 20586 raft_consensus.cc:3037] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:08.637187 20556 raft_consensus.cc:482] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:08.668815 18075 raft_consensus.cc:1226] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 1. Current term is 2. Ops: []
I20260807 03:39:08.646373 20495 raft_consensus.cc:3037] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:08.676558 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:08.677505 18191 raft_consensus.cc:2354] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 6. Current term is 7.
I20260807 03:39:08.618163 20583 consensus_queue.cc:1035] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.045s
I20260807 03:39:08.683354 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:08.692073 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:08.684527 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:08.705686 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:08.706768 18191 raft_consensus.cc:2379] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 3: Already voted for candidate 5c9b5062bdfa46af9396f1a85025a331 in this term.
W20260807 03:39:08.709025 18003 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet c7a9d935105d41829202c4c161ff09e3, TS d9716e2bf06442f5b3b0e81087c53c4d -> TS 5c9b5062bdfa46af9396f1a85025a331 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:08.709438 18003 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet c7a9d935105d41829202c4c161ff09e3, TS d9716e2bf06442f5b3b0e81087c53c4d -> TS 5c9b5062bdfa46af9396f1a85025a331 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:08.681243 17912 catalog_manager.cc:5495] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 3 to 5, leader changed from d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 5 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:39:08.695588 18072 raft_consensus.cc:2454] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:39:08.675374 20556 raft_consensus.cc:504] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:08.718813 20367 raft_consensus.cc:504] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:08.697237 20552 raft_consensus.cc:504] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:08.724066 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:08.723460 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:08.728353 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:08.731694 20595 log.cc:924] Time spent T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.344s user 0.001s sys 0.003s
W20260807 03:39:08.744437 18003 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet fe8bab68c1264fc8a2c4164e0f17f9b6, TS 3c4bceb836db4536a0f255fdf86abc02 -> TS 5c9b5062bdfa46af9396f1a85025a331 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:08.744848 18003 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet fe8bab68c1264fc8a2c4164e0f17f9b6, TS 3c4bceb836db4536a0f255fdf86abc02 -> TS 5c9b5062bdfa46af9396f1a85025a331 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:08.744701 18080 raft_consensus.cc:2454] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 7.
W20260807 03:39:08.747288 18305 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.624s user 0.048s sys 0.063s
W20260807 03:39:08.747656 18305 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.624s user 0.048s sys 0.063s
I20260807 03:39:08.753520 18188 raft_consensus.cc:2354] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 4. Current term is 5.
I20260807 03:39:08.757678 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:08.762310 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:08.748826 17912 catalog_manager.cc:5495] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 2 to 3, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 3 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:08.766568 18073 raft_consensus.cc:2396] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 03:39:08.769076 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:08.769666 18185 raft_consensus.cc:2379] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 3: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
W20260807 03:39:08.778160 18312 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.513s user 0.018s sys 0.042s
I20260807 03:39:08.732023 20124 raft_consensus.cc:504] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:08.780359 18312 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.516s user 0.018s sys 0.042s
I20260807 03:39:08.777446 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:08.776015 20480 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:39:05.657417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45378 (local address 127.15.198.130:37057)
0807 03:39:06.484371 (+826954us) server_negotiation.cc:206] Beginning negotiation
0807 03:39:06.507116 (+ 22745us) server_negotiation.cc:399] Waiting for connection header
0807 03:39:07.711549 (+1204433us) server_negotiation.cc:407] Connection header received
0807 03:39:07.882214 (+170665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:39:07.882254 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:39:07.882585 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:39:07.882851 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:39:08.015435 (+132584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:39:08.458247 (+442812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:39:08.561456 (+103209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45378: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":569304,"thread_start_us":158128,"threads_started":1}
I20260807 03:39:08.791312 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:08.785851 18312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57276 (request call id 729) took 1213 ms. Trace:
I20260807 03:39:08.795819 18312 rpcz_store.cc:276] 0807 03:39:07.567572 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:08.131908 (+564336us) service_pool.cc:227] Handling call
0807 03:39:08.183295 (+ 51387us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:08.216305 (+ 33010us) raft_consensus.cc:1520] Early marking committed up to index 2
0807 03:39:08.229042 (+ 12737us) log.cc:842] Serialized 14 byte log entry
0807 03:39:08.780950 (+551908us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:08.780998 (+ 48us) raft_consensus.cc:1631] Marking committed up to 2
0807 03:39:08.781035 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:08.781097 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:08.781204 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":12215,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":5575,"thread_start_us":516028,"threads_started":1}
I20260807 03:39:08.797616 20552 leader_election.cc:288] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:08.798187 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:08.808959 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:08.809839 18189 raft_consensus.cc:2454] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 2.
I20260807 03:39:08.801891 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:08.850026 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:08.812327 20596 raft_consensus.cc:2781] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:39:08.812863 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:08.813568 18080 raft_consensus.cc:2454] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
I20260807 03:39:08.813817 20556 leader_election.cc:288] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:08.815121 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:08.816793 20367 leader_election.cc:288] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:08.820597 20559 raft_consensus.cc:2229] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:39:08.865859 20559 raft_consensus.cc:2258] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:39:08.802835 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:39:08.869453 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:39:08.871093 20277 raft_consensus.cc:504] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:08.821537 18193 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44660 (request call id 1163) took 1269 ms. Trace:
I20260807 03:39:08.784356 20494 leader_election.cc:288] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
W20260807 03:39:08.873402 20357 consensus_meta.cc:287] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.591s user 0.010s sys 0.002s
I20260807 03:39:08.876629 20559 ts_tablet_manager.cc:1905] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 03:39:08.794934 18072 raft_consensus.cc:2454] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
I20260807 03:39:08.798257 20537 raft_consensus.cc:482] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:08.840368 17912 catalog_manager.cc:5495] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:39:08.861920 20596 raft_consensus.cc:686] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 [term 3 LEADER]: Becoming Leader. State: Replica: 5c9b5062bdfa46af9396f1a85025a331, State: Running, Role: LEADER
I20260807 03:39:08.799656 20124 leader_election.cc:288] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:08.862484 18079 raft_consensus.cc:1261] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Refusing update from remote peer 5c9b5062bdfa46af9396f1a85025a331: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260807 03:39:08.862645 18184 raft_consensus.cc:1261] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260807 03:39:08.873018 19617 consensus_meta.cc:287] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.593s user 0.009s sys 0.001s
I20260807 03:39:08.844758 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:08.872689 18193 rpcz_store.cc:276] 0807 03:39:07.551998 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:07.560446 (+ 8448us) service_pool.cc:227] Handling call
0807 03:39:07.572743 (+ 12297us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 03:39:07.573398 (+ 655us) raft_consensus.cc:1520] Early marking committed up to index 3
0807 03:39:07.857162 (+283764us) log.cc:842] Serialized 14 byte log entry
0807 03:39:08.503190 (+646028us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 03:39:08.538949 (+ 35759us) log.cc:842] Serialized 52 byte log entry
0807 03:39:08.539106 (+ 157us) raft_consensus.cc:1631] Marking committed up to 3
0807 03:39:08.539149 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:08.539231 (+ 82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:08.821266 (+282035us) raft_consensus.cc:1671] finished
0807 03:39:08.821292 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:08.821469 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":33179,"raft.queue_time_us":10516,"raft.run_cpu_time_us":375,"raft.run_wall_time_us":28404,"thread_start_us":600020,"threads_started":1,"wal-append.queue_time_us":933634}
I20260807 03:39:08.932144 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:08.882972 20389 consensus_meta.cc:287] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 0.578s user 0.011s sys 0.001s
I20260807 03:39:08.938820 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:08.947031 18190 raft_consensus.cc:2379] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 5: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:08.946741 20389 raft_consensus.cc:504] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:08.802397 20399 raft_consensus.cc:504] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:08.884425 20537 raft_consensus.cc:3037] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:08.948799 20389 leader_election.cc:288] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:08.949576 18180 raft_consensus.cc:2454] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
I20260807 03:39:08.941846 20397 raft_consensus.cc:504] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:08.952899 20596 consensus_queue.cc:237] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:08.955157 20448 leader_election.cc:288] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163)
I20260807 03:39:08.955994 20042 leader_election.cc:288] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:08.958811 20496 raft_consensus.cc:504] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:08.960244 18071 raft_consensus.cc:2354] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3.
I20260807 03:39:08.961843 19297 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59920 (request call id 35) took 1385 ms. Trace:
I20260807 03:39:08.962194 19297 rpcz_store.cc:276] 0807 03:39:07.576696 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:07.599222 (+ 22526us) service_pool.cc:227] Handling call
0807 03:39:08.961636 (+1362414us) spinlock_profiling.cc:243] Waited 668 ms on lock 0x7b54006504d4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:08.961762 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1535662848}
I20260807 03:39:08.964448 20399 leader_election.cc:288] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:08.944799 20281 raft_consensus.cc:504] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:08.945608 19617 raft_consensus.cc:504] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:08.953662 18293 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57268 (request call id 968) took 2077 ms (client timeout 1557 ms). Trace:
I20260807 03:39:08.942569 18072 raft_consensus.cc:2379] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 6: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:08.966984 18191 raft_consensus.cc:2379] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 4: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:08.941569 18185 raft_consensus.cc:2454] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 5.
I20260807 03:39:08.978482 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:08.979799 18293 rpcz_store.cc:269] 0807 03:39:06.876230 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:06.886327 (+ 10097us) service_pool.cc:227] Handling call
0807 03:39:08.953485 (+2067158us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b54001c9b54. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:08.953601 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3169557888}
I20260807 03:39:08.943150 17912 catalog_manager.cc:5495] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:08.992655 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:08.993322 18071 raft_consensus.cc:2454] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:39:08.993424 20537 raft_consensus.cc:504] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:08.994097 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:08.994534 19617 leader_election.cc:288] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:08.994975 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:08.995440 20537 leader_election.cc:288] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:08.995693 18082 raft_consensus.cc:2379] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 7: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:08.966399 20397 leader_election.cc:288] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:08.959618 20173 raft_consensus.cc:504] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:08.993480 20517 consensus_queue.cc:1035] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
I20260807 03:39:09.004777 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:08.999725 17912 catalog_manager.cc:5495] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 4 to 5, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 5 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:09.005455 20537 raft_consensus.cc:482] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:09.008318 20042 raft_consensus.cc:482] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:09.008425 20537 raft_consensus.cc:504] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:09.009037 20042 raft_consensus.cc:504] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:09.009261 17912 catalog_manager.cc:5495] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:09.010461 20537 leader_election.cc:288] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:09.011061 20042 leader_election.cc:288] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:09.013898 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:09.016736 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:09.017553 18081 raft_consensus.cc:2454] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:39:09.019819 18073 raft_consensus.cc:2379] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 3: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:09.014691 18185 raft_consensus.cc:2454] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 4.
I20260807 03:39:09.010515 17912 catalog_manager.cc:5495] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 1 to 3, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 3 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:09.033663 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:09.033960 17912 catalog_manager.cc:5495] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 1 to 3, leader changed from d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 3 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:39:09.035008 20586 raft_consensus.cc:504] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:09.047189 18187 raft_consensus.cc:1226] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:09.047624 17912 catalog_manager.cc:5495] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:09.051863 18080 raft_consensus.cc:2379] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 3: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:09.072542 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:09.071579 17912 catalog_manager.cc:5495] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 1 to 3, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 3 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:39:09.081084 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:09.083297 20586 leader_election.cc:288] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:09.084127 20281 leader_election.cc:288] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:09.081811 18180 raft_consensus.cc:2454] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 4.
I20260807 03:39:09.091209 18187 raft_consensus.cc:1226] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 4. Current term is 5. Ops: []
I20260807 03:39:09.078666 20277 leader_election.cc:288] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
W20260807 03:39:09.086602 19471 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.644s user 0.012s sys 0.030s
I20260807 03:39:09.080989 20496 leader_election.cc:288] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:09.096684 19471 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.655s user 0.012s sys 0.030s
I20260807 03:39:09.101104 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:09.102415 18180 raft_consensus.cc:2454] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 5.
W20260807 03:39:09.088120 19292 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.589s user 0.000s sys 0.015s
W20260807 03:39:09.115717 19292 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.623s user 0.001s sys 0.015s
W20260807 03:39:09.115798 20503 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.613s user 0.039s sys 0.050s
I20260807 03:39:09.130769 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:09.132140 18180 raft_consensus.cc:2454] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 2.
I20260807 03:39:09.133234 18305 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57276 (request call id 720) took 1698 ms. Trace:
W20260807 03:39:09.127887 20546 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.536s user 0.009s sys 0.009s
W20260807 03:39:09.132990 18141 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.541s user 0.012s sys 0.031s
I20260807 03:39:09.127199 20173 leader_election.cc:288] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:09.145715 18141 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.553s user 0.013s sys 0.036s
I20260807 03:39:09.133543 18305 rpcz_store.cc:276] 0807 03:39:07.435185 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:08.086077 (+650892us) service_pool.cc:227] Handling call
0807 03:39:08.121970 (+ 35893us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:08.122500 (+ 530us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:08.122532 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:39:08.123070 (+ 538us) log.cc:842] Serialized 26 byte log entry
0807 03:39:08.753275 (+630205us) raft_consensus.cc:1631] Marking committed up to 1
0807 03:39:08.753309 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:08.753371 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:09.133051 (+379680us) raft_consensus.cc:1671] finished
0807 03:39:09.133067 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:09.133176 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3875,"raft.queue_time_us":11305,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":22770,"spinlock_wait_cycles":13056,"thread_start_us":624465,"threads_started":1,"wal-append.queue_time_us":953358}
W20260807 03:39:09.147467 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 584745us
W20260807 03:39:09.134372 20503 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.632s user 0.040s sys 0.050s
I20260807 03:39:09.130055 20543 raft_consensus.cc:686] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
W20260807 03:39:09.139736 20546 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.549s user 0.009s sys 0.009s
I20260807 03:39:09.130277 20369 leader_election.cc:288] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
W20260807 03:39:09.156477 18070 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.579s user 0.007s sys 0.014s
I20260807 03:39:09.121047 20495 raft_consensus.cc:504] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:09.180796 18070 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.603s user 0.007s sys 0.014s
I20260807 03:39:09.140646 20397 raft_consensus.cc:482] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 03:39:09.181373 18249 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.532s user 0.012s sys 0.014s
W20260807 03:39:09.183414 18249 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.535s user 0.013s sys 0.014s
W20260807 03:39:09.184834 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 549752us
I20260807 03:39:09.186455 18140 leader_election.cc:302] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:09.188895 18141 leader_election.cc:302] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:09.196427 20399 raft_consensus.cc:2726] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:09.197197 20399 raft_consensus.cc:2781] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 03:39:09.197691 20399 raft_consensus.cc:686] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:09.198177 18141 leader_election.cc:302] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:09.198589 20399 consensus_queue.cc:237] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:09.200273 19292 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59906 (request call id 74) took 1654 ms. Trace:
I20260807 03:39:09.200678 19292 rpcz_store.cc:276] 0807 03:39:07.545849 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:07.546087 (+ 238us) service_pool.cc:227] Handling call
0807 03:39:07.546527 (+ 440us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:07.546854 (+ 327us) raft_consensus.cc:1520] Early marking committed up to index 5
0807 03:39:08.445830 (+898976us) log.cc:842] Serialized 14 byte log entry
0807 03:39:09.146827 (+700997us) log.cc:842] Serialized 14 byte log entry
0807 03:39:09.176706 (+ 29879us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:09.182426 (+ 5720us) raft_consensus.cc:1631] Marking committed up to 5
0807 03:39:09.187667 (+ 5241us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:09.190073 (+ 2406us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:09.200186 (+ 10113us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1170530,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":68,"thread_start_us":1489075,"threads_started":2}
I20260807 03:39:09.202473 20503 raft_consensus.cc:2781] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 03:39:09.203088 20503 raft_consensus.cc:686] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:09.207080 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:09.207827 18073 raft_consensus.cc:2454] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
I20260807 03:39:09.210879 18141 leader_election.cc:398] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021) with higher term. Message: Invalid argument: T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3.
I20260807 03:39:09.214352 18140 leader_election.cc:302] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:09.194005 20517 raft_consensus.cc:482] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:09.205333 20543 consensus_queue.cc:237] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:09.216463 18141 leader_election.cc:302] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:09.218672 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
W20260807 03:39:09.215281 18027 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.627s user 0.003s sys 0.027s
W20260807 03:39:09.220834 18027 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.632s user 0.003s sys 0.028s
I20260807 03:39:09.187916 20397 raft_consensus.cc:504] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:09.215236 19659 raft_consensus.cc:482] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 03:39:09.221138 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 632763us
I20260807 03:39:09.225297 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:09.228444 18081 raft_consensus.cc:3037] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 5
W20260807 03:39:09.215368 19240 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.576s user 0.002s sys 0.028s
I20260807 03:39:09.231274 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:09.212968 20556 consensus_queue.cc:1046] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.012s
I20260807 03:39:09.206202 20503 consensus_queue.cc:237] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:09.235457 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:09.236024 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:09.236279 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:09.214021 20124 consensus_queue.cc:1035] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.014s
W20260807 03:39:09.232424 19240 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.610s user 0.003s sys 0.028s
I20260807 03:39:09.242211 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:09.244310 18309 raft_consensus.cc:3037] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 5
I20260807 03:39:09.253633 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:09.224133 20517 raft_consensus.cc:504] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:09.275028 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:09.257185 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 44696.4us
I20260807 03:39:09.231652 20389 raft_consensus.cc:2726] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:09.230072 20546 raft_consensus.cc:2781] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:39:09.274495 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:09.287837 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:09.292783 18295 raft_consensus.cc:2454] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 6.
I20260807 03:39:09.288615 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:09.293363 18305 raft_consensus.cc:2454] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:39:09.289330 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:09.292963 18073 raft_consensus.cc:2379] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 3: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
W20260807 03:39:09.294953 18075 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.530s user 0.003s sys 0.026s
W20260807 03:39:09.295337 18075 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.531s user 0.003s sys 0.026s
W20260807 03:39:09.290043 18069 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.575s user 0.012s sys 0.020s
I20260807 03:39:09.231897 19659 raft_consensus.cc:504] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:09.308705 18069 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.594s user 0.012s sys 0.021s
W20260807 03:39:09.287412 18183 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.579s user 0.048s sys 0.077s
I20260807 03:39:09.330243 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:09.336323 18300 raft_consensus.cc:2396] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260807 03:39:09.329097 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:09.337215 18080 raft_consensus.cc:2454] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
I20260807 03:39:09.341126 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:09.341806 18305 raft_consensus.cc:3037] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:09.344022 18248 leader_election.cc:302] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
W20260807 03:39:09.307607 20585 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.515s user 0.017s sys 0.013s
W20260807 03:39:09.357939 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 79504.1us
I20260807 03:39:09.292410 20546 raft_consensus.cc:482] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
W20260807 03:39:09.334561 18183 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.641s user 0.048s sys 0.077s
I20260807 03:39:09.359689 18311 raft_consensus.cc:2454] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 5.
I20260807 03:39:09.362617 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:09.294902 19240 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 782674us
I20260807 03:39:09.364583 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:09.372288 18300 raft_consensus.cc:2354] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3.
W20260807 03:39:09.359058 20585 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.570s user 0.017s sys 0.013s
I20260807 03:39:09.374701 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:09.377568 18295 raft_consensus.cc:3037] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:09.375267 18140 leader_election.cc:302] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:09.355715 20495 leader_election.cc:288] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:09.379908 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:09.378643 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:09.379536 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:09.380632 18082 raft_consensus.cc:1226] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260807 03:39:09.363827 20586 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:09.372433 20277 raft_consensus.cc:2781] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:09.392516 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:09.364456 20546 raft_consensus.cc:3037] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:09.403034 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:09.400298 18310 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57276 (request call id 727) took 1894 ms. Trace:
I20260807 03:39:09.408741 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:09.410317 18027 leader_election.cc:398] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Vote denied by peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021) with higher term. Message: Invalid argument: T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 6. Current term is 7.
I20260807 03:39:09.410953 18027 leader_election.cc:401] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260807 03:39:09.410784 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:09.411748 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:09.412381 18082 raft_consensus.cc:2454] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:39:09.413523 18309 raft_consensus.cc:2454] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 5.
I20260807 03:39:09.417423 18309 raft_consensus.cc:1261] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 03:39:09.407840 18310 rpcz_store.cc:276] 0807 03:39:07.509151 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:08.112692 (+603541us) service_pool.cc:227] Handling call
0807 03:39:08.124398 (+ 11706us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:08.124684 (+ 286us) raft_consensus.cc:1520] Early marking committed up to index 3
0807 03:39:08.125043 (+ 359us) log.cc:842] Serialized 14 byte log entry
0807 03:39:09.380585 (+1255542us) log.cc:842] Serialized 14 byte log entry
0807 03:39:09.399928 (+ 19343us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:09.399979 (+ 51us) raft_consensus.cc:1631] Marking committed up to 3
0807 03:39:09.400030 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:09.400097 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:09.400237 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17677,"prepare.queue_time_us":168,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":70787,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":9463,"thread_start_us":1242333,"threads_started":3,"wal-append.queue_time_us":818479}
I20260807 03:39:09.420636 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:09.421672 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:09.391707 20552 raft_consensus.cc:2781] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:09.422649 18082 raft_consensus.cc:2379] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 5: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:09.423463 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
W20260807 03:39:09.425067 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 66529.2us
I20260807 03:39:09.397614 20397 leader_election.cc:288] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:09.402396 20277 raft_consensus.cc:482] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 03:39:09.430393 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 58108.8us
I20260807 03:39:09.423283 18248 leader_election.cc:302] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:09.427995 18082 raft_consensus.cc:2379] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 4: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:09.432953 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:09.427915 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:09.433542 18073 raft_consensus.cc:3037] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:09.435036 18080 raft_consensus.cc:2379] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 5: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:09.436017 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:09.436371 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:09.437117 18080 raft_consensus.cc:2354] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3.
I20260807 03:39:09.437734 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:09.402144 20586 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:09.424328 20552 raft_consensus.cc:482] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:09.442299 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:09.434195 19240 leader_election.cc:302] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:09.455497 18141 leader_election.cc:302] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
W20260807 03:39:09.442397 19238 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.545s user 0.016s sys 0.040s
W20260807 03:39:09.456430 19238 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.701s user 0.016s sys 0.040s
I20260807 03:39:09.456466 18141 leader_election.cc:302] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:09.458132 18141 leader_election.cc:398] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Vote denied by peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021) with higher term. Message: Invalid argument: T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3.
I20260807 03:39:09.458051 19292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:09.458462 18141 leader_election.cc:401] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260807 03:39:09.459391 18141 leader_election.cc:302] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:09.459731 20389 raft_consensus.cc:2781] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:39:09.455518 20552 raft_consensus.cc:3037] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:09.453724 18080 raft_consensus.cc:2454] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:39:09.432507 20277 raft_consensus.cc:3037] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:09.434049 20495 raft_consensus.cc:2726] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021) with higher term. Message: Invalid argument: T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 6. Current term is 7.
I20260807 03:39:09.468055 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:09.468343 18081 raft_consensus.cc:2454] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:09.468740 18082 raft_consensus.cc:3037] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:09.467222 20556 raft_consensus.cc:2781] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:39:09.469697 20556 raft_consensus.cc:482] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:09.470058 20556 raft_consensus.cc:3037] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
W20260807 03:39:09.472817 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 47113us
I20260807 03:39:09.442555 20369 raft_consensus.cc:2781] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:09.474545 18071 raft_consensus.cc:2454] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 6.
I20260807 03:39:09.478317 18183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44660 (request call id 1186) took 1003 ms. Trace:
I20260807 03:39:09.479723 18140 leader_election.cc:302] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:09.485225 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:09.463706 20389 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:09.485855 18293 raft_consensus.cc:1261] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260807 03:39:09.467528 20397 raft_consensus.cc:3037] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:09.440315 20173 consensus_queue.cc:1046] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.040s
I20260807 03:39:09.486227 18293 pending_rounds.cc:76] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49: Aborting all ops after (but not including) 2
I20260807 03:39:09.480240 20369 raft_consensus.cc:482] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:09.493501 20369 raft_consensus.cc:3037] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:09.492177 18293 pending_rounds.cc:97] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260807 03:39:09.492578 20124 raft_consensus.cc:2781] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 03:39:09.485318 19659 leader_election.cc:288] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:09.488667 20389 raft_consensus.cc:3037] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Advancing to term 7
W20260807 03:39:09.493136 20609 log.cc:924] Time spent T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.103s user 0.000s sys 0.000s
I20260807 03:39:09.501065 18293 raft_consensus.cc:2866] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 03:39:09.479792 20596 raft_consensus.cc:2726] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:09.508183 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:09.441514 20042 consensus_queue.cc:1035] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.044s
I20260807 03:39:09.508061 20124 raft_consensus.cc:686] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:09.516800 20124 consensus_queue.cc:237] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:09.521875 18295 raft_consensus.cc:2454] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 5.
W20260807 03:39:09.510926 18027 connection.cc:232] RPC call timeout handler was delayed by -1.0651172411162406s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16910000000000003s for a response
I20260807 03:39:09.531322 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:09.545114 18073 raft_consensus.cc:2454] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 5.
I20260807 03:39:09.521087 20613 raft_consensus.cc:3032] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 LEADER]: Stepping down as leader of term 4
I20260807 03:39:09.553128 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:09.560757 18293 raft_consensus.cc:2454] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:39:09.561959 18297 raft_consensus.cc:2454] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
W20260807 03:39:09.561609 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 86936.4us
I20260807 03:39:09.569681 20277 raft_consensus.cc:504] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:09.571349 20629 log.cc:924] Time spent T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331: Append to log took a long time: real 0.091s user 0.000s sys 0.000s
I20260807 03:39:09.570539 18300 raft_consensus.cc:2454] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:39:09.572175 18027 leader_election.cc:302] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:09.573102 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:09.573264 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:09.574486 18310 raft_consensus.cc:1226] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260807 03:39:09.575282 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:09.577293 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:09.574131 18300 raft_consensus.cc:2454] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3.
I20260807 03:39:09.563371 18249 leader_election.cc:302] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
W20260807 03:39:09.583318 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 109830us
I20260807 03:39:09.584215 18140 leader_election.cc:302] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:09.589696 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:09.589859 18027 leader_election.cc:302] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:09.593201 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:09.594933 20623 raft_consensus.cc:2781] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:39:09.589643 20615 consensus_queue.cc:1035] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20260807 03:39:09.595512 20277 leader_election.cc:288] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:09.595599 20173 consensus_queue.cc:1035] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260807 03:39:09.598481 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:09.599195 18295 raft_consensus.cc:3037] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:09.601675 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:09.602442 18298 raft_consensus.cc:2379] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 6: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:09.603149 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:09.603240 20649 consensus_queue.cc:1035] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:39:09.603830 18298 raft_consensus.cc:2379] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 5: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:09.604791 20173 raft_consensus.cc:2726] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:09.480629 18183 rpcz_store.cc:276] 0807 03:39:08.474581 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:08.475718 (+ 1137us) service_pool.cc:227] Handling call
0807 03:39:08.544320 (+ 68602us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:08.564234 (+ 19914us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:08.568957 (+ 4723us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:39:08.648032 (+ 79075us) log.cc:842] Serialized 26 byte log entry
0807 03:39:09.375997 (+727965us) raft_consensus.cc:1631] Marking committed up to 2
0807 03:39:09.428584 (+ 52587us) log.cc:842] Serialized 14 byte log entry
0807 03:39:09.440286 (+ 11702us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:09.452065 (+ 11779us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:09.478070 (+ 26005us) raft_consensus.cc:1671] finished
0807 03:39:09.478103 (+ 33us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:09.478258 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21769,"prepare.queue_time_us":40928,"prepare.run_wall_time_us":3862,"raft.queue_time_us":13634,"raft.run_cpu_time_us":267,"raft.run_wall_time_us":18163,"spinlock_wait_cycles":7046528,"thread_start_us":671911,"threads_started":1,"wal-append.queue_time_us":772392}
I20260807 03:39:09.605338 18027 leader_election.cc:398] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Vote denied by peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057) with higher term. Message: Invalid argument: T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 6. Current term is 7.
I20260807 03:39:09.606020 18305 raft_consensus.cc:2454] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:39:09.606274 18027 leader_election.cc:302] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:09.606909 18249 leader_election.cc:302] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:09.607829 20495 raft_consensus.cc:2726] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 03:39:09.608845 20651 raft_consensus.cc:2781] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:09.609351 20651 raft_consensus.cc:482] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:09.607174 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:09.609810 20651 raft_consensus.cc:3037] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:09.610081 18311 raft_consensus.cc:3037] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:09.613451 18249 leader_election.cc:302] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:09.614337 20367 raft_consensus.cc:3032] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 03:39:09.614781 18249 leader_election.cc:302] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:09.614969 20367 raft_consensus.cc:729] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:09.560477 20613 raft_consensus.cc:729] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:09.616067 20367 consensus_queue.cc:260] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:09.616276 20613 consensus_queue.cc:260] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:09.617283 20367 raft_consensus.cc:3037] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:09.617872 20613 raft_consensus.cc:3037] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:09.595547 20623 raft_consensus.cc:482] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:09.620430 20623 raft_consensus.cc:3037] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:09.621057 18249 leader_election.cc:302] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:09.625860 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:09.627625 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:09.628465 20586 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:09.630092 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:09.629891 20546 raft_consensus.cc:504] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:09.630788 18180 raft_consensus.cc:2379] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 5: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:09.631335 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:09.631898 18187 raft_consensus.cc:2454] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
I20260807 03:39:09.614430 20618 raft_consensus.cc:2781] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:39:09.607471 18027 leader_election.cc:302] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:09.634380 20618 raft_consensus.cc:482] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:09.636847 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:09.643661 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:09.636919 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:09.645540 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:09.646019 18193 raft_consensus.cc:2379] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 3: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:09.587203 20494 raft_consensus.cc:2741] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
W20260807 03:39:09.647043 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 174160us
I20260807 03:39:09.647187 18297 raft_consensus.cc:2379] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 3: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:09.668152 18069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33268 (request call id 968) took 1109 ms. Trace:
I20260807 03:39:09.648720 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:09.669231 18293 raft_consensus.cc:3037] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 5
W20260807 03:39:09.669673 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 62304.5us
W20260807 03:39:09.670498 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 207364us
I20260807 03:39:09.674372 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:09.675648 18188 raft_consensus.cc:3037] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:09.676512 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:09.586216 20559 ts_tablet_manager.cc:1918] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 03:39:09.678676 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:09.679286 18193 raft_consensus.cc:2454] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
I20260807 03:39:09.632021 20546 leader_election.cc:288] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:09.668614 18069 rpcz_store.cc:276] 0807 03:39:08.558698 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:08.639903 (+ 81205us) service_pool.cc:227] Handling call
0807 03:39:08.656010 (+ 16107us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:08.687250 (+ 31240us) raft_consensus.cc:1520] Early marking committed up to index 3
0807 03:39:09.334380 (+647130us) log.cc:842] Serialized 14 byte log entry
0807 03:39:09.638652 (+304272us) log.cc:842] Serialized 14 byte log entry
0807 03:39:09.667620 (+ 28968us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:09.667670 (+ 50us) raft_consensus.cc:1631] Marking committed up to 3
0807 03:39:09.667706 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:09.667773 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:09.668081 (+ 308us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7550,"prepare.queue_time_us":505782,"prepare.run_cpu_time_us":25,"prepare.run_wall_time_us":25,"thread_start_us":887845,"threads_started":2}
I20260807 03:39:09.676066 18248 leader_election.cc:302] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:09.681387 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:09.681533 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:09.636843 20618 raft_consensus.cc:3037] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:09.682188 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:09.683291 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 99214.1us
I20260807 03:39:09.681108 18141 leader_election.cc:302] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:09.669068 20495 raft_consensus.cc:2726] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:09.632791 20552 raft_consensus.cc:504] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:09.694721 18027 leader_election.cc:302] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:09.693547 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:09.696902 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:09.697013 18141 leader_election.cc:302] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:09.699205 18141 leader_election.cc:302] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:09.674264 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:09.692517 18249 leader_election.cc:302] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:09.676306 20649 raft_consensus.cc:482] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:09.678514 20448 raft_consensus.cc:482] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:09.704716 18248 leader_election.cc:302] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:09.714602 18295 raft_consensus.cc:2454] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3.
I20260807 03:39:09.715649 20653 raft_consensus.cc:2781] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 03:39:09.715942 20495 raft_consensus.cc:2726] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:09.691973 20503 raft_consensus.cc:482] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:09.695953 20559 log.cc:1196] T 1b214306f53244f8b6ff2a1a9019abd1 P 3c4bceb836db4536a0f255fdf86abc02: Deleting WAL directory at /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786073845599630-16154-0/minicluster-data/ts-1-root/wals/1b214306f53244f8b6ff2a1a9019abd1
I20260807 03:39:09.696017 20511 raft_consensus.cc:482] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:09.722185 18248 leader_election.cc:302] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:09.720053 18069 raft_consensus.cc:1261] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Refusing update from remote peer 5c9b5062bdfa46af9396f1a85025a331: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260807 03:39:09.702369 20634 log.cc:924] Time spent T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.142s user 0.001s sys 0.000s
I20260807 03:39:09.726334 18027 leader_election.cc:302] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:09.726661 18311 raft_consensus.cc:2454] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3.
I20260807 03:39:09.727548 18248 leader_election.cc:302] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:09.702657 20277 raft_consensus.cc:2781] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:09.698954 20546 raft_consensus.cc:2781] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:09.711632 18305 raft_consensus.cc:2454] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:39:09.707552 18184 raft_consensus.cc:3037] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
W20260807 03:39:09.720677 20632 log.cc:924] Time spent T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.068s user 0.000s sys 0.001s
I20260807 03:39:09.736963 20552 leader_election.cc:288] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:09.741153 20543 raft_consensus.cc:2781] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:09.741621 20543 raft_consensus.cc:482] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:09.742059 20543 raft_consensus.cc:3037] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
W20260807 03:39:09.748301 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 49166.1us
I20260807 03:39:09.751920 18141 leader_election.cc:302] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:09.753257 20124 raft_consensus.cc:2781] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:39:09.753808 20124 raft_consensus.cc:482] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:09.754309 20124 raft_consensus.cc:3037] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:09.725791 19471 raft_consensus.cc:2781] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:09.730455 19743 raft_consensus.cc:2741] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260807 03:39:09.731768 20156 raft_consensus.cc:2726] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:09.770109 18141 leader_election.cc:302] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:09.771442 18140 leader_election.cc:302] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:09.730865 20511 raft_consensus.cc:504] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:09.774397 20648 raft_consensus.cc:2726] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:09.742944 20281 raft_consensus.cc:2726] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:09.775957 20511 leader_election.cc:288] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:09.777545 18027 leader_election.cc:302] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:09.778085 20658 raft_consensus.cc:2781] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader election won for term 2
W20260807 03:39:09.779291 20654 log.cc:924] Time spent T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20260807 03:39:09.779677 18027 leader_election.cc:398] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 election: Vote denied by peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057) with higher term. Message: Invalid argument: T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 4. Current term is 5.
I20260807 03:39:09.771121 20615 raft_consensus.cc:2781] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 03:39:09.781409 20042 raft_consensus.cc:2726] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:09.784487 20620 raft_consensus.cc:2741] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260807 03:39:09.785940 20667 raft_consensus.cc:2781] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:39:09.786448 20667 raft_consensus.cc:686] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 3 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:09.787163 20559 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44650 (request call id 110) took 3148 ms. Trace:
I20260807 03:39:09.717044 20448 raft_consensus.cc:504] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:09.780019 18027 leader_election.cc:401] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260807 03:39:09.732757 20546 raft_consensus.cc:482] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:09.800271 18188 raft_consensus.cc:2454] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:09.803508 18140 leader_election.cc:302] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:09.787568 20559 rpcz_store.cc:276] 0807 03:39:06.638252 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:06.702193 (+ 63941us) service_pool.cc:227] Handling call
0807 03:39:07.361334 (+659141us) ts_tablet_manager.cc:1145] Deleting tablet 1b214306f53244f8b6ff2a1a9019abd1
0807 03:39:07.361508 (+ 174us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0807 03:39:09.110401 (+1748893us) tablet_metadata.cc:641] Metadata flushed
0807 03:39:09.544243 (+433842us) tablet_metadata.cc:641] Metadata flushed
0807 03:39:09.787024 (+242781us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1022699,"tablet-delete.queue_time_us":544388,"thread_start_us":420193,"threads_started":1}
I20260807 03:39:09.806341 20389 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:09.773058 20585 log.cc:924] Time spent T 00000000000000000000000000000000 P 8da2065a77024f35bdfc0dc3adb83f69: Append to log took a long time: real 0.069s user 0.002s sys 0.000s
I20260807 03:39:09.808881 18027 leader_election.cc:302] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:09.708045 20649 raft_consensus.cc:504] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:09.770533 19471 raft_consensus.cc:482] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:09.819813 18183 raft_consensus.cc:2454] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:39:09.824048 18189 raft_consensus.cc:2354] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 6. Current term is 7.
W20260807 03:39:09.825150 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 97706.1us
I20260807 03:39:09.826156 18248 leader_election.cc:302] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:09.787389 20667 consensus_queue.cc:237] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:09.732805 20503 raft_consensus.cc:504] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:09.834360 20658 raft_consensus.cc:686] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 2 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:09.834555 18248 leader_election.cc:302] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:09.835443 18027 leader_election.cc:302] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:09.835291 20658 consensus_queue.cc:237] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:09.836740 18027 leader_election.cc:302] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:09.809706 20546 raft_consensus.cc:3037] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:09.807824 18293 raft_consensus.cc:2454] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 5.
I20260807 03:39:09.838009 18248 leader_election.cc:302] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
W20260807 03:39:09.827657 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 135238us
I20260807 03:39:09.842082 18248 leader_election.cc:302] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:09.842813 20281 raft_consensus.cc:2781] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:39:09.843158 20281 raft_consensus.cc:482] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:09.843493 20281 raft_consensus.cc:3037] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:09.822705 20042 raft_consensus.cc:2726] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057) with higher term. Message: Invalid argument: T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 4. Current term is 5.
I20260807 03:39:09.825685 20367 raft_consensus.cc:2726] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:09.803272 20615 raft_consensus.cc:686] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:09.827944 19471 raft_consensus.cc:3037] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:09.825379 20496 raft_consensus.cc:2726] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:09.852234 18082 raft_consensus.cc:2454] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:39:09.836237 20495 raft_consensus.cc:2781] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:09.854290 18249 leader_election.cc:302] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:09.825898 20653 raft_consensus.cc:686] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:09.839130 19617 raft_consensus.cc:2726] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260807 03:39:09.858597 18248 leader_election.cc:302] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:09.848172 20123 raft_consensus.cc:2781] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:39:09.867774 18248 leader_election.cc:302] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:09.869130 18248 leader_election.cc:302] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:09.851207 20511 raft_consensus.cc:3037] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:09.832880 20556 raft_consensus.cc:504] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:09.837843 20586 consensus_queue.cc:1035] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.023s
W20260807 03:39:09.887092 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 102457us
I20260807 03:39:09.890024 17899 catalog_manager.cc:4841] TS 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057): tablet 1b214306f53244f8b6ff2a1a9019abd1 (table test-workload [id=677ca051b50345e183d60556fc42291d]) successfully deleted
W20260807 03:39:09.890719 17899 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 50417.3us
I20260807 03:39:09.892364 20448 leader_election.cc:288] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:09.893666 20621 raft_consensus.cc:2726] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 03:39:09.894233 20556 leader_election.cc:288] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:09.886428 19617 raft_consensus.cc:2741] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260807 03:39:09.896704 20678 raft_consensus.cc:2781] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:39:09.897197 20678 raft_consensus.cc:482] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:09.897537 20678 raft_consensus.cc:3037] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:09.897709 20517 leader_election.cc:288] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:09.898319 20517 consensus_queue.cc:1035] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 03:39:09.898480 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:09.901393 20156 raft_consensus.cc:2781] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 03:39:09.901897 20156 raft_consensus.cc:686] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:09.863030 20367 consensus_queue.cc:1046] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
I20260807 03:39:09.903160 20367 raft_consensus.cc:3032] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:39:09.903502 20367 raft_consensus.cc:729] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:09.904321 20367 consensus_queue.cc:260] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:09.905555 20367 raft_consensus.cc:3037] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:09.860898 20042 raft_consensus.cc:2741] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260807 03:39:09.857687 20495 raft_consensus.cc:482] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:09.862629 20173 raft_consensus.cc:2726] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 03:39:09.902761 20156 consensus_queue.cc:237] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:09.911199 18141 leader_election.cc:302] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:09.864677 20123 raft_consensus.cc:482] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:09.889075 20559 tablet_replica.cc:330] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02: stopping tablet replica
W20260807 03:39:09.915896 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 79384.2us
I20260807 03:39:09.917718 18027 leader_election.cc:302] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:09.861821 20124 raft_consensus.cc:504] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:09.922847 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:09.910069 20495 raft_consensus.cc:3037] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:09.922899 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:09.916307 20123 raft_consensus.cc:3037] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:09.938848 18184 raft_consensus.cc:2454] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 6.
I20260807 03:39:09.910997 20369 raft_consensus.cc:504] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:09.945539 20559 raft_consensus.cc:2229] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Raft consensus shutting down.
W20260807 03:39:09.948374 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 99022.4us
I20260807 03:39:09.996845 18249 leader_election.cc:302] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:09.952746 20389 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:09.956785 20649 leader_election.cc:288] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:09.957358 20677 raft_consensus.cc:2726] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:09.965902 18027 leader_election.cc:302] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
W20260807 03:39:10.014474 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 48694.6us
I20260807 03:39:09.968452 18312 raft_consensus.cc:2354] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 4. Current term is 5.
I20260807 03:39:09.888965 20615 consensus_queue.cc:237] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:09.977134 20503 leader_election.cc:288] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:09.981374 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:10.019429 18298 raft_consensus.cc:3037] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:09.982568 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:10.020226 18293 raft_consensus.cc:2454] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:39:09.985219 20498 consensus_queue.cc:1046] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 03:39:10.008522 20369 leader_election.cc:288] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:10.021260 18141 leader_election.cc:302] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:10.022583 18027 leader_election.cc:302] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:09.928610 20688 raft_consensus.cc:2781] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:39:09.991504 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:09.961230 20586 raft_consensus.cc:2726] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 03:39:10.028438 18249 leader_election.cc:302] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:10.021962 20648 raft_consensus.cc:2741] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260807 03:39:10.026116 20494 raft_consensus.cc:2726] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:10.034837 18070 raft_consensus.cc:2354] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3.
I20260807 03:39:10.044646 18027 leader_election.cc:302] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:10.039574 20369 raft_consensus.cc:2726] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:10.050242 18249 leader_election.cc:302] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:10.050803 18027 leader_election.cc:302] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:10.060215 18249 leader_election.cc:398] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 election: Vote denied by peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057) with higher term. Message: Invalid argument: T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 6. Current term is 7.
I20260807 03:39:10.060611 18249 leader_election.cc:401] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
W20260807 03:39:10.061292 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 257903us
W20260807 03:39:10.061410 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 192350us
I20260807 03:39:10.039696 20496 raft_consensus.cc:482] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:10.044081 20688 raft_consensus.cc:686] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:10.043821 20583 raft_consensus.cc:2781] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:39:10.065757 20682 raft_consensus.cc:2781] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 03:39:10.059369 20494 raft_consensus.cc:2726] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:10.068153 20124 leader_election.cc:288] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:10.068646 18193 raft_consensus.cc:1261] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Refusing update from remote peer 5c9b5062bdfa46af9396f1a85025a331: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260807 03:39:09.936955 20653 consensus_queue.cc:237] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:10.070191 20559 raft_consensus.cc:2258] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 03:39:10.087011 20583 raft_consensus.cc:482] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:10.100059 20583 raft_consensus.cc:3037] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:10.101562 20559 ts_tablet_manager.cc:1905] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 03:39:10.102187 20682 raft_consensus.cc:686] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 6 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:10.103098 20682 consensus_queue.cc:237] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:10.069862 20623 raft_consensus.cc:504] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:10.105571 20623 leader_election.cc:288] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:10.072605 20511 raft_consensus.cc:2726] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:10.106581 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:10.073153 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:10.074134 20543 raft_consensus.cc:504] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:10.080919 20585 log.cc:924] Time spent T 00000000000000000000000000000000 P 8da2065a77024f35bdfc0dc3adb83f69: Append to log took a long time: real 0.051s user 0.002s sys 0.000s
I20260807 03:39:10.082435 20613 raft_consensus.cc:3032] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:39:10.074090 17920 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35098 (request call id 24) took 3384 ms. Trace:
I20260807 03:39:10.034641 20686 raft_consensus.cc:2781] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:10.085309 18140 leader_election.cc:302] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:10.085799 18027 leader_election.cc:302] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:10.084605 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:10.056567 20498 raft_consensus.cc:2781] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:10.093098 20688 consensus_queue.cc:237] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:10.106676 18310 raft_consensus.cc:1226] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 1. Current term is 2. Ops: []
I20260807 03:39:10.033355 20691 consensus_queue.cc:1035] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.050s
I20260807 03:39:10.108574 18311 raft_consensus.cc:1261] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 03:39:10.113678 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:10.127970 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
W20260807 03:39:10.130473 20399 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.508s user 0.000s sys 0.012s
I20260807 03:39:10.141316 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:10.141343 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 55614.8us
I20260807 03:39:10.177397 18297 raft_consensus.cc:1261] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Refusing update from remote peer 5c9b5062bdfa46af9396f1a85025a331: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260807 03:39:10.143102 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:10.179435 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 94173.6us
I20260807 03:39:10.145213 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:10.180369 18070 raft_consensus.cc:3037] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:10.181048 18295 raft_consensus.cc:3037] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:10.182374 18027 leader_election.cc:302] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:10.183197 18082 raft_consensus.cc:3037] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 5
I20260807 03:39:10.141368 18190 raft_consensus.cc:2362] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 6. Re-sending same reply.
I20260807 03:39:10.145895 18293 raft_consensus.cc:1226] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:10.186965 18140 leader_election.cc:302] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:10.178332 18248 leader_election.cc:302] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:10.086090 20496 raft_consensus.cc:504] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:10.190728 18071 raft_consensus.cc:1261] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260807 03:39:10.192018 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 131941us
I20260807 03:39:10.196704 20496 leader_election.cc:288] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:10.196923 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:10.197642 18185 raft_consensus.cc:2379] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 7: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:10.199878 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:10.201416 19471 raft_consensus.cc:504] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:10.160346 20042 raft_consensus.cc:2781] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:39:10.203271 19471 leader_election.cc:288] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:10.203516 20701 raft_consensus.cc:2726] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260807 03:39:10.165864 20651 consensus_meta.cc:287] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 0.555s user 0.010s sys 0.003s
I20260807 03:39:10.204272 20694 raft_consensus.cc:2781] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:39:10.204370 20651 raft_consensus.cc:504] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:10.205426 20277 raft_consensus.cc:482] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:10.205312 20546 raft_consensus.cc:504] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:10.205951 20277 raft_consensus.cc:3037] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:10.206532 20651 leader_election.cc:288] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:10.207267 20546 leader_election.cc:288] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:10.125845 20281 raft_consensus.cc:504] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:10.208953 20543 leader_election.cc:288] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:10.209858 20281 leader_election.cc:288] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
W20260807 03:39:10.210896 20397 consensus_meta.cc:287] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 0.720s user 0.008s sys 0.001s
I20260807 03:39:10.211303 20397 raft_consensus.cc:2726] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021) with higher term. Message: Invalid argument: T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3.
I20260807 03:39:10.204843 20694 raft_consensus.cc:686] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:10.199532 20626 consensus_queue.cc:1035] T af8877c3fbba4ab18765832f1390921f P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
W20260807 03:39:10.188421 20399 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.566s user 0.000s sys 0.014s
I20260807 03:39:10.128240 20613 raft_consensus.cc:729] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:10.211064 20583 raft_consensus.cc:504] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:10.193887 19240 leader_election.cc:302] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
W20260807 03:39:10.277428 19240 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 83630.9us
I20260807 03:39:10.140488 20686 raft_consensus.cc:482] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:10.278096 20596 raft_consensus.cc:2726] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:10.278312 20686 raft_consensus.cc:3037] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:10.272022 20623 raft_consensus.cc:482] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:10.280133 20623 raft_consensus.cc:504] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:10.281976 20623 leader_election.cc:288] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:10.260692 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
W20260807 03:39:10.283290 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 65074.6us
I20260807 03:39:10.242139 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:10.211850 18188 raft_consensus.cc:1226] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 4. Current term is 5. Ops: []
I20260807 03:39:10.288391 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:10.290884 20495 raft_consensus.cc:504] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:10.291771 18298 raft_consensus.cc:2454] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 7.
I20260807 03:39:10.294150 18070 raft_consensus.cc:2454] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
I20260807 03:39:10.292011 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:10.300127 18295 raft_consensus.cc:2454] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 6.
I20260807 03:39:10.185751 20498 raft_consensus.cc:482] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:10.329154 20498 raft_consensus.cc:3037] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:10.329893 20495 leader_election.cc:288] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:10.317530 18248 leader_election.cc:302] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:10.323230 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:10.315936 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:10.332468 18072 raft_consensus.cc:2454] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
I20260807 03:39:10.307657 18082 raft_consensus.cc:2454] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:10.176941 20699 raft_consensus.cc:482] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:10.314275 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:10.335589 18248 leader_election.cc:302] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:10.237932 20042 raft_consensus.cc:686] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:10.320379 20448 raft_consensus.cc:2726] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057) with higher term. Message: Invalid argument: T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 6. Current term is 7.
I20260807 03:39:10.320962 20613 consensus_queue.cc:260] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:10.322216 20694 consensus_queue.cc:237] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:10.128439 17920 rpcz_store.cc:276] 0807 03:39:06.689565 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:06.689733 (+ 168us) service_pool.cc:227] Handling call
0807 03:39:10.073854 (+3384121us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 03:39:07.291266 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 03:39:07.293698 (+ 2432us) write_op.cc:432] Acquiring schema lock in shared mode
0807 03:39:07.293764 (+ 66us) write_op.cc:435] Acquired schema lock
0807 03:39:07.293815 (+ 51us) tablet.cc:660] Decoding operations
0807 03:39:07.294502 (+ 687us) write_op.cc:620] Acquiring the partition lock for write op
0807 03:39:07.294629 (+ 127us) write_op.cc:641] Partition lock acquired for write op
0807 03:39:07.294694 (+ 65us) tablet.cc:683] Acquiring locks for 1 operations
0807 03:39:07.295152 (+ 458us) tablet.cc:699] Row locks acquired
0807 03:39:07.295188 (+ 36us) write_op.cc:260] PREPARE: finished
0807 03:39:07.295534 (+ 346us) write_op.cc:270] Start()
0807 03:39:07.295670 (+ 136us) write_op.cc:276] Timestamp: P: 1786073947295441 usec, L: 0
0807 03:39:07.295709 (+ 39us) op_driver.cc:331] REPLICATION: starting
0807 03:39:07.329164 (+ 33455us) log.cc:842] Serialized 602 byte log entry
0807 03:39:09.551974 (+2222810us) op_driver.cc:444] REPLICATION: finished
0807 03:39:09.788609 (+236635us) write_op.cc:301] APPLY: starting
0807 03:39:09.809455 (+ 20846us) tablet.cc:1365] starting BulkCheckPresence
0807 03:39:09.839449 (+ 29994us) tablet.cc:1368] finished BulkCheckPresence
0807 03:39:09.862968 (+ 23519us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 03:39:09.894513 (+ 31545us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 03:39:09.894842 (+ 329us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0807 03:39:09.894915 (+ 73us) write_op.cc:312] APPLY: finished
0807 03:39:09.895215 (+ 300us) log.cc:842] Serialized 22 byte log entry
0807 03:39:09.895426 (+ 211us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:39:09.895517 (+ 91us) write_op.cc:454] Released schema lock
0807 03:39:09.895584 (+ 67us) write_op.cc:341] FINISH: Updating metrics
0807 03:39:09.895968 (+ 384us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:39:09.896007 (+ 39us) write_op.cc:454] Released schema lock
0807 03:39:10.080807 (+184800us) spinlock_profiling.cc:243] Waited 2.24 ms on lock 0x7b2000136778. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe991e54f11 00007fe991e76b09 00007fe991e41c43 00007fe991e40f36 00007fe991e40b58 00007fe991e4a551 00007fe991e4a509 00007fe991e4a499 00007fe991e4a461 00007fe991e4975d 00007fe9984fb204
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":145907,"apply.run_cpu_time_us":2872,"apply.run_wall_time_us":144234,"num_ops":1,"prepare.queue_time_us":343931,"prepare.run_cpu_time_us":35632,"prepare.run_wall_time_us":117959,"raft.queue_time_us":768209,"raft.run_cpu_time_us":22509,"raft.run_wall_time_us":209640,"replication_time_us":2242014,"spinlock_wait_cycles":5147904,"thread_start_us":1017775,"threads_started":4,"wal-append.queue_time_us":877562}]]}
I20260807 03:39:10.323230 19292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:10.324671 18141 leader_election.cc:302] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:10.347335 18141 leader_election.cc:302] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:10.278101 20583 leader_election.cc:288] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:10.191888 20667 consensus_queue.cc:1035] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.147s
I20260807 03:39:10.251121 20369 consensus_queue.cc:1035] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.078s
I20260807 03:39:10.328522 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:10.338634 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:10.358222 18079 raft_consensus.cc:3037] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:10.337373 20623 raft_consensus.cc:2781] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 03:39:10.159534 20123 raft_consensus.cc:504] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:10.359213 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:10.333722 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:39:10.317746 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 130829us
W20260807 03:39:10.341059 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 158747us
I20260807 03:39:10.343513 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:10.217118 20552 raft_consensus.cc:2726] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:10.349709 18248 leader_election.cc:302] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:10.349844 18075 raft_consensus.cc:2454] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:39:10.328009 18187 raft_consensus.cc:2454] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:39:10.189453 20678 raft_consensus.cc:504] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:10.330739 20620 raft_consensus.cc:2726] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:10.353600 18189 raft_consensus.cc:1261] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 03:39:10.353940 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:10.357012 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:10.360960 18183 raft_consensus.cc:2379] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 3: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:10.368204 18140 leader_election.cc:302] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:10.384321 18191 raft_consensus.cc:3037] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:10.335106 20367 consensus_queue.cc:1046] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20260807 03:39:10.361816 18027 leader_election.cc:302] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:10.385706 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:10.352506 19659 raft_consensus.cc:2781] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:10.348773 20596 consensus_queue.cc:1035] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20260807 03:39:10.359144 20613 raft_consensus.cc:3037] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:10.397142 18027 leader_election.cc:398] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Vote denied by peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021) with higher term. Message: Invalid argument: T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 4. Current term is 5.
I20260807 03:39:10.493983 18027 leader_election.cc:401] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260807 03:39:10.423018 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:10.423277 18183 raft_consensus.cc:1226] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 3. Current term is 4. Ops: []
W20260807 03:39:10.473634 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 112912us
I20260807 03:39:10.472008 20281 raft_consensus.cc:2781] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 03:39:10.496099 18140 leader_election.cc:302] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:10.480209 20277 raft_consensus.cc:504] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:10.358660 20651 raft_consensus.cc:2781] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 03:39:10.358424 20688 raft_consensus.cc:2781] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 03:39:10.488376 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:10.490131 18189 raft_consensus.cc:1226] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 1. Current term is 2. Ops: []
I20260807 03:39:10.423164 18312 raft_consensus.cc:1261] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260807 03:39:10.497618 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 100679us
W20260807 03:39:10.464293 20618 consensus_meta.cc:287] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.763s user 0.010s sys 0.000s
I20260807 03:39:10.499486 18079 raft_consensus.cc:2454] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:10.386890 20623 raft_consensus.cc:686] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:10.344894 20699 raft_consensus.cc:504] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:10.507664 20618 raft_consensus.cc:504] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:10.508306 20586 consensus_queue.cc:1046] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:39:10.509186 18305 raft_consensus.cc:2379] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 3: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:10.509536 20586 raft_consensus.cc:3032] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:39:10.509982 20586 raft_consensus.cc:729] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:10.510528 20618 leader_election.cc:288] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:10.510896 20586 consensus_queue.cc:260] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:10.462062 20546 raft_consensus.cc:2726] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260807 03:39:10.513162 20712 log.cc:924] Time spent T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.148s user 0.001s sys 0.000s
I20260807 03:39:10.516206 18140 leader_election.cc:302] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:10.452271 20173 raft_consensus.cc:2781] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election won for term 5
W20260807 03:39:10.471374 19240 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 119087us
I20260807 03:39:10.470908 20042 consensus_queue.cc:237] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:10.462688 20367 raft_consensus.cc:3032] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:39:10.517912 20586 raft_consensus.cc:3037] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:10.504058 18188 raft_consensus.cc:2379] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 3: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:10.396832 19659 raft_consensus.cc:482] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:10.500492 20494 raft_consensus.cc:2726] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021) with higher term. Message: Invalid argument: T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 4. Current term is 5.
I20260807 03:39:10.557006 18140 leader_election.cc:302] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:10.556973 18191 raft_consensus.cc:2454] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:10.520677 20281 raft_consensus.cc:686] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:10.529150 20667 raft_consensus.cc:2726] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:10.555852 20546 raft_consensus.cc:2781] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:39:10.583945 20678 leader_election.cc:288] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:10.584509 20546 raft_consensus.cc:482] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
W20260807 03:39:10.584894 20715 log.cc:924] Time spent T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.099s user 0.003s sys 0.000s
I20260807 03:39:10.584980 20546 raft_consensus.cc:3037] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:10.529000 20367 raft_consensus.cc:729] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:10.529362 20688 raft_consensus.cc:686] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 7 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:10.556357 20651 raft_consensus.cc:686] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:10.579828 18305 raft_consensus.cc:1226] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:10.582310 20123 leader_election.cc:288] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:10.560034 20173 raft_consensus.cc:686] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:10.587836 18140 leader_election.cc:302] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:10.575840 19659 raft_consensus.cc:3037] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
W20260807 03:39:10.607146 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 257511us
I20260807 03:39:10.609611 18248 leader_election.cc:302] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:10.617235 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:10.617652 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:10.620574 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:10.622948 18248 leader_election.cc:302] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:10.624346 18249 leader_election.cc:302] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
W20260807 03:39:10.610297 20616 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.001s user 0.003s sys 0.037s
W20260807 03:39:10.625768 20616 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.017s user 0.003s sys 0.038s
I20260807 03:39:10.590658 20701 raft_consensus.cc:2781] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:39:10.627166 18309 raft_consensus.cc:1226] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 5. Current term is 6. Ops: [3.2-5.3]
I20260807 03:39:10.632345 18140 leader_election.cc:398] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Vote denied by peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163) with higher term. Message: Invalid argument: T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3.
I20260807 03:39:10.633482 18140 leader_election.cc:302] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:10.634797 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:10.635776 18305 raft_consensus.cc:2379] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 7: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:10.588011 20686 raft_consensus.cc:504] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:10.642107 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:10.626418 19471 raft_consensus.cc:2726] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:10.623337 20397 raft_consensus.cc:2781] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:10.611099 20399 consensus_queue.cc:1035] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20260807 03:39:10.668675 18248 leader_election.cc:302] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:10.630256 20701 raft_consensus.cc:482] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:10.676515 18140 leader_election.cc:302] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:10.680891 18140 leader_election.cc:302] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:10.681658 20543 raft_consensus.cc:2781] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:39:10.684834 18140 leader_election.cc:398] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Vote denied by peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163) with higher term. Message: Invalid argument: T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 6. Current term is 7.
I20260807 03:39:10.685107 18140 leader_election.cc:401] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260807 03:39:10.673347 20556 raft_consensus.cc:2726] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 03:39:10.678858 20552 raft_consensus.cc:2726] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 03:39:10.676133 20397 raft_consensus.cc:482] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
W20260807 03:39:10.687160 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 62871.8us
I20260807 03:39:10.689952 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:10.697750 18080 raft_consensus.cc:3037] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:10.676127 20281 consensus_queue.cc:237] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:10.693311 18140 leader_election.cc:302] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:10.682122 20648 raft_consensus.cc:2726] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:10.721045 20686 leader_election.cc:288] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:10.695349 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:10.679641 20701 raft_consensus.cc:3037] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:10.684687 20277 leader_election.cc:288] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:10.687489 19471 raft_consensus.cc:2781] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 03:39:10.688028 20615 raft_consensus.cc:2726] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:10.724714 18140 leader_election.cc:302] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:10.725571 20686 raft_consensus.cc:2781] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 03:39:10.726042 20686 raft_consensus.cc:686] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:10.726970 20686 consensus_queue.cc:237] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:10.728243 20552 raft_consensus.cc:2726] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:10.689422 20173 consensus_queue.cc:237] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:10.731609 18140 leader_election.cc:398] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Vote denied by peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163) with higher term. Message: Invalid argument: T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3.
I20260807 03:39:10.665650 20367 consensus_queue.cc:260] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:10.732434 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:10.732682 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:10.733069 18069 raft_consensus.cc:3037] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:10.692692 20543 raft_consensus.cc:686] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:10.683655 20688 consensus_queue.cc:237] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:10.732420 18140 leader_election.cc:302] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:10.616848 20623 consensus_queue.cc:237] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:10.691820 20124 raft_consensus.cc:3037] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:10.746120 18080 raft_consensus.cc:2454] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:39:10.700862 20397 raft_consensus.cc:3037] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:10.681380 20498 raft_consensus.cc:504] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:10.756253 20653 raft_consensus.cc:2726] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 03:39:10.761672 19659 raft_consensus.cc:504] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:10.762943 20717 log.cc:924] Time spent T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20260807 03:39:10.677824 20651 consensus_queue.cc:237] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:10.764432 20543 consensus_queue.cc:237] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:10.732800 20367 raft_consensus.cc:3037] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:10.718950 20699 leader_election.cc:288] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:10.755386 20621 consensus_queue.cc:1046] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
I20260807 03:39:10.806823 18073 raft_consensus.cc:2379] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 4: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:10.780929 19659 leader_election.cc:288] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:10.777948 19471 raft_consensus.cc:686] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:10.782552 18190 raft_consensus.cc:1226] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 4. Current term is 5. Ops: []
I20260807 03:39:10.783497 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:10.809983 18191 raft_consensus.cc:2379] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 3: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:10.784621 18069 raft_consensus.cc:2454] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:39:10.785353 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:10.788546 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:10.788645 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:10.811463 18185 raft_consensus.cc:3037] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:10.763600 18140 leader_election.cc:302] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:10.811431 18193 raft_consensus.cc:2454] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
W20260807 03:39:10.815850 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 52360.7us
I20260807 03:39:10.808936 20546 raft_consensus.cc:504] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:10.819352 18184 raft_consensus.cc:1226] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:10.820406 18140 leader_election.cc:302] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:10.822162 18140 leader_election.cc:302] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:10.823625 18140 leader_election.cc:302] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:10.817641 20667 raft_consensus.cc:2726] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 03:39:10.840976 20498 leader_election.cc:288] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:10.844256 20399 raft_consensus.cc:2726] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:10.835392 20653 raft_consensus.cc:2726] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:10.834702 20552 raft_consensus.cc:3037] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:10.854342 18140 leader_election.cc:398] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Vote denied by peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163) with higher term. Message: Invalid argument: T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3.
W20260807 03:39:10.857002 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 106708us
I20260807 03:39:10.872665 18140 leader_election.cc:302] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:10.875624 18185 raft_consensus.cc:2454] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:10.881373 20559 ts_tablet_manager.cc:1918] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 03:39:10.884670 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:10.885219 18072 raft_consensus.cc:3037] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:10.886683 20546 leader_election.cc:288] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:10.888108 18080 raft_consensus.cc:1261] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260807 03:39:10.890286 18300 raft_consensus.cc:1226] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 4. Current term is 5. Ops: []
I20260807 03:39:10.890581 18081 raft_consensus.cc:1261] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260807 03:39:10.891368 18309 raft_consensus.cc:1226] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3. Ops: []
I20260807 03:39:10.891094 20397 raft_consensus.cc:504] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:10.898886 20667 consensus_queue.cc:1046] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:39:10.899570 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:10.899857 20667 consensus_queue.cc:1046] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:39:10.901265 20667 raft_consensus.cc:3032] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 03:39:10.901692 20667 raft_consensus.cc:729] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:10.902994 20667 consensus_queue.cc:260] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:10.901189 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:10.904407 19471 consensus_queue.cc:237] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:10.905471 20667 raft_consensus.cc:3037] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:10.876549 18140 leader_election.cc:302] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
W20260807 03:39:10.910163 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 127289us
I20260807 03:39:10.913952 18072 raft_consensus.cc:2454] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:10.915208 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:10.881934 20559 log.cc:1196] T 071a939ead594d38851e3b554bfc4d24 P 3c4bceb836db4536a0f255fdf86abc02: Deleting WAL directory at /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786073845599630-16154-0/minicluster-data/ts-1-root/wals/071a939ead594d38851e3b554bfc4d24
I20260807 03:39:10.921007 18312 raft_consensus.cc:1261] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:39:10.880057 20653 raft_consensus.cc:2726] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:10.924058 18249 leader_election.cc:302] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
W20260807 03:39:10.928980 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 52509.1us
I20260807 03:39:10.857061 20701 raft_consensus.cc:504] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:10.877048 20498 consensus_queue.cc:1046] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20260807 03:39:10.936218 18140 leader_election.cc:398] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Vote denied by peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163) with higher term. Message: Invalid argument: T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3.
I20260807 03:39:10.944139 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:10.946843 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:10.948241 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:10.948786 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:10.949623 18071 raft_consensus.cc:2354] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3.
I20260807 03:39:10.949738 18082 raft_consensus.cc:1261] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260807 03:39:10.950825 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:10.953043 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:10.953727 18071 raft_consensus.cc:3037] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:10.961601 18075 raft_consensus.cc:1226] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 4. Current term is 5. Ops: []
I20260807 03:39:10.961544 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:10.962260 18082 raft_consensus.cc:3037] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:10.962478 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:10.965500 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:10.966480 18079 raft_consensus.cc:2379] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 3: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:10.967552 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:10.968300 18069 raft_consensus.cc:3037] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:10.934890 20556 raft_consensus.cc:2781] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:10.979576 18082 raft_consensus.cc:2454] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 6.
W20260807 03:39:10.980370 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 61078.3us
W20260807 03:39:10.981312 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 57333.6us
I20260807 03:39:10.983596 18080 raft_consensus.cc:1261] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:39:10.985562 18080 raft_consensus.cc:1226] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3. Ops: []
I20260807 03:39:10.985656 18082 raft_consensus.cc:1261] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260807 03:39:10.986927 18080 raft_consensus.cc:1226] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 1. Current term is 2. Ops: []
I20260807 03:39:10.987176 18082 raft_consensus.cc:1226] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3. Ops: []
I20260807 03:39:10.993623 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:10.992084 18248 leader_election.cc:302] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
W20260807 03:39:10.994983 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 70792.4us
I20260807 03:39:10.996341 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:11.005668 18069 raft_consensus.cc:2454] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 7.
I20260807 03:39:10.937467 20688 consensus_queue.cc:1035] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.055s
I20260807 03:39:10.982386 20556 raft_consensus.cc:482] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:11.009739 18249 leader_election.cc:302] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:10.991273 20648 raft_consensus.cc:3037] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:10.984393 20399 raft_consensus.cc:482] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:11.018043 20556 raft_consensus.cc:3037] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:10.994972 20615 raft_consensus.cc:482] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:11.007148 20559 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44650 (request call id 109) took 4465 ms. Trace:
I20260807 03:39:11.095408 20397 leader_election.cc:288] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:11.098495 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 88835.2us
I20260807 03:39:11.099664 18297 raft_consensus.cc:1226] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3. Ops: []
W20260807 03:39:11.101786 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 109793us
I20260807 03:39:11.102671 18248 leader_election.cc:302] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:11.114346 20651 consensus_queue.cc:1046] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:39:11.115278 20546 raft_consensus.cc:3032] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 03:39:11.115708 20546 raft_consensus.cc:729] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
W20260807 03:39:11.116009 20742 log.cc:924] Time spent T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
I20260807 03:39:11.094444 20559 rpcz_store.cc:276] 0807 03:39:06.576884 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:06.667284 (+ 90400us) service_pool.cc:227] Handling call
0807 03:39:09.851182 (+3183898us) ts_tablet_manager.cc:1145] Deleting tablet 071a939ead594d38851e3b554bfc4d24
0807 03:39:09.861890 (+ 10708us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0807 03:39:10.638161 (+776271us) tablet_metadata.cc:641] Metadata flushed
0807 03:39:10.862786 (+224625us) tablet_metadata.cc:641] Metadata flushed
0807 03:39:10.993054 (+130268us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2983430}
W20260807 03:39:11.107892 20741 log.cc:924] Time spent T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.085s user 0.000s sys 0.001s
I20260807 03:39:11.111358 20496 raft_consensus.cc:482] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:11.159260 20496 raft_consensus.cc:504] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:11.162155 20496 leader_election.cc:288] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:11.100832 20620 raft_consensus.cc:2726] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260807 03:39:11.109289 20727 log.cc:924] Time spent T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.100s user 0.000s sys 0.001s
I20260807 03:39:11.163141 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:11.168784 20735 log.cc:924] Time spent T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20260807 03:39:11.173570 20613 raft_consensus.cc:482] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:11.174135 20613 raft_consensus.cc:504] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:11.176472 20701 leader_election.cc:288] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:11.111989 20124 raft_consensus.cc:2726] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163) with higher term. Message: Invalid argument: T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 6. Current term is 7.
I20260807 03:39:11.177743 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:11.116544 20546 consensus_queue.cc:260] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:11.017238 20543 consensus_queue.cc:1035] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.095s
W20260807 03:39:11.146727 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 44141.8us
W20260807 03:39:11.146860 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 46562us
I20260807 03:39:11.187469 18249 leader_election.cc:302] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:11.112193 19471 raft_consensus.cc:2781] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:39:11.113736 20746 raft_consensus.cc:482] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 03:39:11.144137 20712 log.cc:924] Time spent T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.124s user 0.001s sys 0.009s
I20260807 03:39:11.025409 20399 raft_consensus.cc:504] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:11.188208 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:11.198391 18187 raft_consensus.cc:1261] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260807 03:39:11.198958 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:11.199757 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:11.199829 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:11.200470 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:11.200759 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:11.201344 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:11.213107 18193 raft_consensus.cc:2379] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 4: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:11.104247 20615 raft_consensus.cc:504] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:11.182502 20699 raft_consensus.cc:2781] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:39:11.215826 18190 raft_consensus.cc:2354] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 4. Current term is 5.
W20260807 03:39:11.158490 20717 log.cc:924] Time spent T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.052s user 0.000s sys 0.001s
I20260807 03:39:11.180076 20546 raft_consensus.cc:3037] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:11.218708 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:11.176051 20613 leader_election.cc:288] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:11.212627 18191 raft_consensus.cc:2454] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 4.
I20260807 03:39:11.223634 18311 raft_consensus.cc:1261] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:39:11.226090 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:11.231472 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:11.234403 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:11.238469 18309 raft_consensus.cc:1261] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:39:11.208273 20688 raft_consensus.cc:482] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:11.240576 20688 raft_consensus.cc:504] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:11.243147 20688 leader_election.cc:288] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:11.244406 20496 raft_consensus.cc:482] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:11.244908 20496 raft_consensus.cc:504] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:11.248595 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:11.249109 20173 raft_consensus.cc:3032] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Stepping down as leader of term 5
I20260807 03:39:11.249588 20173 raft_consensus.cc:729] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:11.249982 20399 leader_election.cc:288] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:11.253433 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:11.255477 18071 raft_consensus.cc:2454] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:39:11.260577 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:11.270121 20713 consensus_queue.cc:1035] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 03:39:11.270179 20703 consensus_queue.cc:1035] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 03:39:11.215096 17899 catalog_manager.cc:4841] TS 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057): tablet 071a939ead594d38851e3b554bfc4d24 (table test-workload [id=677ca051b50345e183d60556fc42291d]) successfully deleted
W20260807 03:39:11.272907 17899 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 66214us
I20260807 03:39:11.273798 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:11.274538 20733 consensus_queue.cc:1046] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.054s
I20260807 03:39:11.276467 18311 raft_consensus.cc:1261] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260807 03:39:11.214609 20746 raft_consensus.cc:504] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:11.278291 20615 leader_election.cc:288] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:11.279094 20746 leader_election.cc:288] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:11.279448 20552 raft_consensus.cc:2726] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:11.283020 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:11.234081 19471 raft_consensus.cc:686] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
W20260807 03:39:11.279384 20743 log.cc:924] Time spent T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.088s user 0.000s sys 0.000s
I20260807 03:39:11.252291 20699 raft_consensus.cc:686] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:11.294958 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:11.296799 18184 raft_consensus.cc:2379] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 3: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:11.297379 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:11.298275 18311 raft_consensus.cc:2379] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 3: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:11.250613 20173 consensus_queue.cc:260] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:11.300166 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:11.284258 20721 raft_consensus.cc:2726] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 03:39:11.251917 20613 consensus_queue.cc:1035] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.041s
I20260807 03:39:11.293886 18003 placement_policy_util.cc:244] tablet 071a939ead594d38851e3b554bfc4d24: not considering replicas for movement since the tablet's status is 'UNAVAILABLE'
I20260807 03:39:11.311532 18297 raft_consensus.cc:2379] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 2: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:11.280825 20678 consensus_queue.cc:1046] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20260807 03:39:11.317349 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:11.318259 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:11.319069 18190 raft_consensus.cc:3037] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:11.306838 20173 raft_consensus.cc:3037] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:11.291326 20713 consensus_queue.cc:1035] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20260807 03:39:11.355546 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:11.374097 20699 consensus_queue.cc:237] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:11.372654 19471 consensus_queue.cc:237] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:11.418097 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 183758us
I20260807 03:39:11.421046 18183 raft_consensus.cc:2454] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 5.
I20260807 03:39:11.421482 18249 leader_election.cc:302] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:11.422752 18190 raft_consensus.cc:2454] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3.
I20260807 03:39:11.426746 18293 raft_consensus.cc:1261] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20260807 03:39:11.427968 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 194938us
I20260807 03:39:11.428617 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:11.428572 18190 raft_consensus.cc:1226] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260807 03:39:11.435402 18183 raft_consensus.cc:1226] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 4. Current term is 5. Ops: [2.2-4.3]
I20260807 03:39:11.435827 18190 raft_consensus.cc:1226] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260807 03:39:11.424160 20613 raft_consensus.cc:482] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:11.428356 20737 raft_consensus.cc:2781] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:39:11.437233 18249 leader_election.cc:302] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:11.443620 20648 raft_consensus.cc:2726] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 03:39:11.444912 20737 raft_consensus.cc:686] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:11.445737 20737 consensus_queue.cc:237] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:11.447535 18180 raft_consensus.cc:1226] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 4. Current term is 5. Ops: []
I20260807 03:39:11.440789 20684 consensus_queue.cc:1035] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.012s
I20260807 03:39:11.450891 17912 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39494 (request call id 135) took 5442 ms. Trace:
I20260807 03:39:11.451092 18191 raft_consensus.cc:1226] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:11.451494 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:11.438460 20613 raft_consensus.cc:504] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:11.452147 18185 raft_consensus.cc:1261] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:39:11.428776 20556 raft_consensus.cc:504] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:11.452158 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:11.452839 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:11.453513 20649 raft_consensus.cc:2781] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:39:11.454795 20543 consensus_queue.cc:1046] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:39:11.437024 20123 raft_consensus.cc:3032] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 03:39:11.456190 18249 leader_election.cc:302] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:11.458716 18184 raft_consensus.cc:2396] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 03:39:11.320753 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260807 03:39:11.462004 20732 log.cc:924] Time spent T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.610s user 0.003s sys 0.007s
I20260807 03:39:11.462563 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:11.469326 20613 leader_election.cc:288] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:11.469906 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
W20260807 03:39:11.470093 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 182792us
I20260807 03:39:11.470216 20496 leader_election.cc:288] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:11.469909 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:11.471045 20556 leader_election.cc:288] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:11.471637 18187 raft_consensus.cc:2379] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 4: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:11.472061 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:11.472488 18188 raft_consensus.cc:2379] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 3: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:11.473765 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:11.455331 17914 catalog_manager.cc:5495] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 1 to 3, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 3 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:11.497044 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:11.501122 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:11.501729 18071 raft_consensus.cc:2454] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:39:11.501812 18305 raft_consensus.cc:1226] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 4. Current term is 5. Ops: []
I20260807 03:39:11.503589 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:11.503907 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:11.504364 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:11.508625 18183 raft_consensus.cc:1261] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260807 03:39:11.510417 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 68783.7us
I20260807 03:39:11.511314 18248 leader_election.cc:302] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:11.507493 17914 catalog_manager.cc:5495] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 2 to 1, leader changed from <none> to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 1 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 03:39:11.520490 18082 raft_consensus.cc:1261] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20260807 03:39:11.520967 18082 pending_rounds.cc:76] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d: Aborting all ops after (but not including) 2
I20260807 03:39:11.521203 18082 pending_rounds.cc:97] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260807 03:39:11.521456 18082 raft_consensus.cc:2866] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 03:39:11.524336 20277 raft_consensus.cc:2781] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 03:39:11.525630 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:11.524325 20761 consensus_queue.cc:1035] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 03:39:11.472265 20649 raft_consensus.cc:482] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:11.483620 20123 raft_consensus.cc:729] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:11.526772 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:11.633569 18079 raft_consensus.cc:1261] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260807 03:39:11.634454 18082 raft_consensus.cc:1226] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:11.634325 18071 raft_consensus.cc:1226] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 1. Current term is 2. Ops: []
I20260807 03:39:11.465363 17912 rpcz_store.cc:276] 0807 03:39:06.008053 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:06.017507 (+ 9454us) service_pool.cc:227] Handling call
0807 03:39:11.450792 (+5433285us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 03:39:09.596302 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 03:39:09.596879 (+ 577us) write_op.cc:432] Acquiring schema lock in shared mode
0807 03:39:09.596923 (+ 44us) write_op.cc:435] Acquired schema lock
0807 03:39:09.596945 (+ 22us) tablet.cc:660] Decoding operations
0807 03:39:09.601432 (+ 4487us) write_op.cc:620] Acquiring the partition lock for write op
0807 03:39:09.601491 (+ 59us) write_op.cc:641] Partition lock acquired for write op
0807 03:39:09.601514 (+ 23us) tablet.cc:683] Acquiring locks for 18 operations
0807 03:39:09.602700 (+ 1186us) tablet.cc:699] Row locks acquired
0807 03:39:09.602719 (+ 19us) write_op.cc:260] PREPARE: finished
0807 03:39:09.602848 (+ 129us) write_op.cc:270] Start()
0807 03:39:09.602973 (+ 125us) write_op.cc:276] Timestamp: P: 1786073949602810 usec, L: 0
0807 03:39:09.602994 (+ 21us) op_driver.cc:331] REPLICATION: starting
0807 03:39:09.703804 (+100810us) log.cc:842] Serialized 7191 byte log entry
0807 03:39:09.909676 (+205872us) op_driver.cc:444] REPLICATION: finished
0807 03:39:09.930845 (+ 21169us) write_op.cc:301] APPLY: starting
0807 03:39:09.932919 (+ 2074us) tablet.cc:1365] starting BulkCheckPresence
0807 03:39:09.971358 (+ 38439us) tablet.cc:1368] finished BulkCheckPresence
0807 03:39:09.971507 (+ 149us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 03:39:09.973448 (+ 1941us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 03:39:09.973766 (+ 318us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=18
0807 03:39:09.973810 (+ 44us) write_op.cc:312] APPLY: finished
0807 03:39:09.974127 (+ 317us) log.cc:842] Serialized 124 byte log entry
0807 03:39:09.974267 (+ 140us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:39:09.974417 (+ 150us) write_op.cc:454] Released schema lock
0807 03:39:09.974466 (+ 49us) write_op.cc:341] FINISH: Updating metrics
0807 03:39:09.974714 (+ 248us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:39:09.974739 (+ 25us) write_op.cc:454] Released schema lock
0807 03:39:10.075600 (+100861us) spinlock_profiling.cc:243] Waited 76.8 ms on lock 0x7b54002611d4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bb198 00007fe995bceade 00007fe995bce2b8 00007fe995bd2ba1 00007fe995bd2b59 00007fe995bd2ae9 00007fe995bd2ab1 00007fe995bd1dad 00007fe9984fb204 00007fe9984fb039
Metrics: {"mutex_wait_us":117572,"reported_tablets":26,"child_traces":[["op",{"apply.queue_time_us":4065,"apply.run_cpu_time_us":4490,"apply.run_wall_time_us":46802,"num_ops":18,"prepare.queue_time_us":157297,"prepare.run_cpu_time_us":7623,"prepare.run_wall_time_us":479472,"replication_time_us":271825,"spinlock_wait_cycles":176646016,"thread_start_us":115379,"threads_started":1}]]}
I20260807 03:39:11.637585 18193 raft_consensus.cc:2454] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 4.
I20260807 03:39:11.637771 18140 leader_election.cc:302] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:11.558533 18069 raft_consensus.cc:1261] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20260807 03:39:11.638759 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 182658us
I20260807 03:39:11.640879 18140 leader_election.cc:398] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Vote denied by peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163) with higher term. Message: Invalid argument: T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3.
I20260807 03:39:11.642042 18140 leader_election.cc:302] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:11.530071 20277 raft_consensus.cc:686] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:11.642210 20648 raft_consensus.cc:2781] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election won for term 5
W20260807 03:39:11.637574 18027 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.15.198.130:37057 timed out after 1.839s (SENT) [suppressed 16 similar messages]
I20260807 03:39:11.642228 20556 consensus_queue.cc:1035] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260807 03:39:11.664403 18140 leader_election.cc:302] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:11.662837 18309 raft_consensus.cc:1226] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 5. Current term is 6. Ops: [3.2-5.3]
I20260807 03:39:11.551983 20613 raft_consensus.cc:482] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:11.664387 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:11.665992 18193 raft_consensus.cc:1261] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260807 03:39:11.668572 18140 leader_election.cc:302] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:11.675930 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:11.680892 18191 raft_consensus.cc:1261] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260807 03:39:11.642922 20649 raft_consensus.cc:3037] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:11.641415 20546 raft_consensus.cc:2726] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 03:39:11.678220 18027 leader_election.cc:302] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:11.685997 18140 leader_election.cc:302] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:11.642482 20737 raft_consensus.cc:482] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:11.686955 18185 raft_consensus.cc:1226] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 4. Current term is 5. Ops: []
I20260807 03:39:11.686154 18188 raft_consensus.cc:1261] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260807 03:39:11.684128 18305 raft_consensus.cc:1226] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 2. Current term is 3. Ops: []
I20260807 03:39:11.667202 20124 raft_consensus.cc:2726] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 03:39:11.686378 18027 leader_election.cc:302] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:11.689829 18185 raft_consensus.cc:1226] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 2. Current term is 3. Ops: []
I20260807 03:39:11.689721 20751 consensus_queue.cc:1035] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 03:39:11.690222 20173 raft_consensus.cc:2726] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:11.670616 20653 raft_consensus.cc:2726] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:11.670346 20448 consensus_queue.cc:1046] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260807 03:39:11.683502 20586 consensus_queue.cc:1035] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20260807 03:39:11.691583 18188 raft_consensus.cc:3032] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 03:39:11.694403 18027 leader_election.cc:302] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:11.714902 18188 raft_consensus.cc:729] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:11.715672 18027 leader_election.cc:302] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:11.716251 18140 leader_election.cc:302] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:11.717162 18027 leader_election.cc:302] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:11.690461 20688 raft_consensus.cc:2781] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 03:39:11.670428 20746 raft_consensus.cc:482] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:11.725533 18140 leader_election.cc:302] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:11.688683 20648 raft_consensus.cc:686] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:11.684269 20615 raft_consensus.cc:2726] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 03:39:11.736148 18027 leader_election.cc:302] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:11.737286 20494 consensus_queue.cc:1046] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:39:11.738233 20494 raft_consensus.cc:2726] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 03:39:11.738871 20494 raft_consensus.cc:3032] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 LEADER]: Stepping down as leader of term 2
I20260807 03:39:11.739187 20494 raft_consensus.cc:729] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
W20260807 03:39:11.683933 20686 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.954s user 0.004s sys 0.007s
I20260807 03:39:11.733013 18140 leader_election.cc:302] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:11.698333 20694 raft_consensus.cc:2726] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:11.739915 20494 consensus_queue.cc:260] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:11.740231 20583 consensus_queue.cc:1046] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260807 03:39:11.740480 20686 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.010s user 0.004s sys 0.007s
I20260807 03:39:11.741075 20694 raft_consensus.cc:3032] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 LEADER]: Stepping down as leader of term 4
I20260807 03:39:11.741664 20694 raft_consensus.cc:729] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:11.741155 20494 raft_consensus.cc:3037] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:11.741914 20691 consensus_queue.cc:1046] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 03:39:11.679015 20613 raft_consensus.cc:504] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:11.752592 18140 leader_election.cc:302] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:11.732563 18188 consensus_queue.cc:260] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:11.710157 20546 consensus_queue.cc:1035] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.051s
I20260807 03:39:11.734715 20124 raft_consensus.cc:2726] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 03:39:11.772993 18140 leader_election.cc:302] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:11.754184 20653 raft_consensus.cc:2726] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 03:39:11.735069 20173 raft_consensus.cc:2726] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 03:39:11.734613 20765 raft_consensus.cc:2726] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:11.696738 20737 raft_consensus.cc:504] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:11.753089 20713 raft_consensus.cc:2726] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 03:39:11.742565 20694 consensus_queue.cc:260] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:11.695936 20123 consensus_queue.cc:260] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:11.735080 20658 consensus_queue.cc:1046] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
I20260807 03:39:11.786214 18140 leader_election.cc:302] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:11.734790 20746 raft_consensus.cc:504] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:11.772389 20615 raft_consensus.cc:2781] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:39:11.761266 20583 consensus_queue.cc:1035] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20260807 03:39:11.827354 18311 raft_consensus.cc:1261] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260807 03:39:11.827567 18072 raft_consensus.cc:1261] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:39:11.782783 20751 raft_consensus.cc:2726] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 03:39:11.786947 20124 raft_consensus.cc:2726] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260807 03:39:11.857769 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 71657.8us
I20260807 03:39:11.860312 18140 leader_election.cc:302] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:11.761576 20277 consensus_queue.cc:237] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:11.871690 18003 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica 3c4bceb836db4536a0f255fdf86abc02 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:11.866117 20688 raft_consensus.cc:686] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:11.874958 20688 consensus_queue.cc:237] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:11.888068 18003 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 349190ae153f46c9a99c29df20734d3f, TS d9716e2bf06442f5b3b0e81087c53c4d -> TS 5c9b5062bdfa46af9396f1a85025a331 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:11.889319 18003 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 349190ae153f46c9a99c29df20734d3f, TS d9716e2bf06442f5b3b0e81087c53c4d -> TS 5c9b5062bdfa46af9396f1a85025a331 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:11.825141 20765 consensus_queue.cc:1046] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.059s
I20260807 03:39:11.871232 20124 raft_consensus.cc:2781] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260807 03:39:11.919082 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 147327us
I20260807 03:39:11.821504 20648 consensus_queue.cc:237] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:11.887434 20586 raft_consensus.cc:482] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 03:39:11.939967 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 54335.6us
I20260807 03:39:11.940812 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:11.908867 20713 raft_consensus.cc:482] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:11.943462 18189 raft_consensus.cc:2354] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 5. Current term is 6.
I20260807 03:39:11.908808 20658 raft_consensus.cc:2726] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:11.922116 20546 raft_consensus.cc:482] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:11.923272 20124 raft_consensus.cc:482] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:11.941515 20583 raft_consensus.cc:482] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:11.908392 20543 consensus_queue.cc:1046] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.036s
I20260807 03:39:11.883373 20789 consensus_queue.cc:1035] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.062s
W20260807 03:39:11.955830 20793 log.cc:924] Time spent T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.069s user 0.001s sys 0.000s
I20260807 03:39:11.947484 20713 raft_consensus.cc:504] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:11.959187 20713 leader_election.cc:288] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:11.944696 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 84482us
I20260807 03:39:11.948853 20124 raft_consensus.cc:3037] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:11.963135 18140 leader_election.cc:302] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:11.964552 18140 leader_election.cc:302] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
W20260807 03:39:12.021329 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 47275.4us
I20260807 03:39:11.919448 20795 consensus_queue.cc:1046] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
I20260807 03:39:11.944262 20586 raft_consensus.cc:504] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:12.036540 18071 raft_consensus.cc:1226] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 4. Current term is 5. Ops: []
I20260807 03:39:12.037544 18027 leader_election.cc:302] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:11.953411 20613 leader_election.cc:288] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:12.041772 18027 leader_election.cc:302] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:11.962550 20737 leader_election.cc:288] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:11.949527 20546 raft_consensus.cc:504] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:11.962437 20615 raft_consensus.cc:686] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:11.965456 20658 raft_consensus.cc:2726] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 03:39:12.047161 18075 raft_consensus.cc:1261] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 4. (term mismatch)
I20260807 03:39:12.049032 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:12.050329 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:12.050915 18027 leader_election.cc:302] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:12.052078 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:11.953634 20583 raft_consensus.cc:504] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:12.053643 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:12.058081 18185 raft_consensus.cc:2379] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 4: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:12.059038 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:12.061045 18075 pending_rounds.cc:76] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d: Aborting all ops after (but not including) 3
I20260807 03:39:12.061400 18075 pending_rounds.cc:97] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20260807 03:39:12.061177 20799 raft_consensus.cc:2726] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:12.061725 18075 raft_consensus.cc:2866] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 03:39:12.042029 20746 leader_election.cc:288] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:12.063612 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
W20260807 03:39:12.064399 20809 log.cc:924] Time spent T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.108s user 0.001s sys 0.000s
I20260807 03:39:12.064291 20649 raft_consensus.cc:504] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:12.065739 20792 log.cc:924] Time spent T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.154s user 0.000s sys 0.001s
I20260807 03:39:12.045770 20713 raft_consensus.cc:2726] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260807 03:39:12.067714 20730 log.cc:924] Time spent T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 1.221s user 0.000s sys 0.003s
I20260807 03:39:12.049832 20765 raft_consensus.cc:2726] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 03:39:12.070825 18071 raft_consensus.cc:2454] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:39:12.059810 18185 raft_consensus.cc:2454] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 7.
I20260807 03:39:12.080236 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:12.071519 18072 raft_consensus.cc:3037] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:12.080917 18187 raft_consensus.cc:2354] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 4. Current term is 5.
I20260807 03:39:12.072468 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:12.078613 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:39:12.083781 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 134266us
I20260807 03:39:12.031692 20556 consensus_queue.cc:1035] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20260807 03:39:12.089358 18027 leader_election.cc:302] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:12.039088 20688 consensus_queue.cc:1035] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.031s
I20260807 03:39:12.092371 18071 raft_consensus.cc:3037] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:12.095649 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:12.096366 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:12.097432 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:12.097965 18187 raft_consensus.cc:2454] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 5.
I20260807 03:39:12.099040 18027 leader_election.cc:302] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:12.100396 18027 leader_election.cc:302] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:12.100543 18071 raft_consensus.cc:1261] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260807 03:39:12.105672 18140 leader_election.cc:302] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:12.109041 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:12.109236 18081 raft_consensus.cc:1226] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:12.104321 20746 consensus_queue.cc:1046] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260807 03:39:12.116750 18073 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33254 (request call id 1147) took 1796 ms (client timeout 1777 ms). Trace:
I20260807 03:39:12.103169 20583 leader_election.cc:288] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:12.157217 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:12.121461 20615 consensus_queue.cc:237] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:12.104948 20718 raft_consensus.cc:3032] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 03:39:12.093106 20795 consensus_queue.cc:1035] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.034s
I20260807 03:39:12.117728 20802 consensus_queue.cc:1035] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.010s
I20260807 03:39:12.114432 20804 raft_consensus.cc:2726] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:12.091698 20691 raft_consensus.cc:2726] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:12.113456 20799 consensus_queue.cc:1035] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 03:39:12.262563 18193 raft_consensus.cc:1226] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 3. Current term is 4. Ops: []
W20260807 03:39:12.263060 18073 rpcz_store.cc:269] 0807 03:39:10.311462 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:10.314772 (+ 3310us) service_pool.cc:227] Handling call
0807 03:39:12.108191 (+1793419us) spinlock_profiling.cc:243] Waited 448 ms on lock 0x7b5400341a54. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:12.108338 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1031290624}
I20260807 03:39:12.267339 18072 raft_consensus.cc:2454] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:39:12.264838 20649 leader_election.cc:288] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:12.275198 18187 raft_consensus.cc:1261] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260807 03:39:12.281057 18071 raft_consensus.cc:1261] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260807 03:39:12.289696 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 177275us
I20260807 03:39:12.265231 20713 raft_consensus.cc:2781] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader election won for term 4
I20260807 03:39:12.271031 20511 raft_consensus.cc:482] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:12.289191 20816 raft_consensus.cc:482] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 03:39:12.296232 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 195886us
I20260807 03:39:12.269068 20718 raft_consensus.cc:729] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:12.307595 18298 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57268 (request call id 1037) took 1710 ms. Trace:
I20260807 03:39:12.309082 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:12.307337 18027 leader_election.cc:398] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Vote denied by peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057) with higher term. Message: Invalid argument: T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 5. Current term is 6.
I20260807 03:39:12.320006 18298 rpcz_store.cc:276] 0807 03:39:10.597520 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:10.611260 (+ 13740us) service_pool.cc:227] Handling call
0807 03:39:10.611528 (+ 268us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:10.618122 (+ 6594us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:10.618152 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:39:10.632749 (+ 14597us) log.cc:842] Serialized 26 byte log entry
0807 03:39:10.803105 (+170356us) raft_consensus.cc:1631] Marking committed up to 3
0807 03:39:10.803157 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:10.803247 (+ 90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:12.067084 (+1263837us) spinlock_profiling.cc:243] Waited 63.5 ms on lock 0x7b44000232cc. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f1a9a95 00007fe98f1aa4b2 00007fe991e52511 00007fe991e4de1a 00007fe991e41bee 00007fe991e40f36 00007fe991e40b58 00007fe991e4a551 00007fe991e4a509 00007fe991e4a499
0807 03:39:12.293664 (+226580us) raft_consensus.cc:1671] finished
0807 03:39:12.293686 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:12.307529 (+ 13843us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34759,"raft.queue_time_us":10490,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":4176,"spinlock_wait_cycles":146107648,"thread_start_us":118433,"threads_started":1,"wal-append.queue_time_us":173994}
I20260807 03:39:12.321646 18027 leader_election.cc:401] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260807 03:39:12.298805 18187 raft_consensus.cc:1261] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260807 03:39:12.323351 18187 pending_rounds.cc:76] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02: Aborting all ops after (but not including) 1
I20260807 03:39:12.323678 18187 pending_rounds.cc:97] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260807 03:39:12.323961 18187 raft_consensus.cc:2866] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 03:39:12.331698 18027 leader_election.cc:302] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:12.313339 19743 raft_consensus.cc:3032] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 03:39:12.303853 20751 raft_consensus.cc:482] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:12.333529 19743 raft_consensus.cc:729] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:12.310571 20511 raft_consensus.cc:504] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:12.333887 20751 raft_consensus.cc:504] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:12.293555 20613 raft_consensus.cc:2726] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:12.347770 20446 consensus_queue.cc:1035] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20260807 03:39:12.348466 20511 leader_election.cc:288] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:12.334306 19743 consensus_queue.cc:260] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:12.349630 19743 raft_consensus.cc:3037] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:12.313311 20546 leader_election.cc:288] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:12.329751 20713 raft_consensus.cc:686] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:12.367393 20813 raft_consensus.cc:482] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:12.368180 20751 leader_election.cc:288] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:12.367889 20813 raft_consensus.cc:504] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:12.361559 20583 raft_consensus.cc:3037] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:12.357115 20779 raft_consensus.cc:482] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:12.360338 20586 leader_election.cc:288] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:12.372454 20779 raft_consensus.cc:504] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
W20260807 03:39:12.374125 20494 consensus_meta.cc:287] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 0.632s user 0.009s sys 0.006s
I20260807 03:39:12.374558 20779 leader_election.cc:288] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:12.376765 20713 consensus_queue.cc:237] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:12.387714 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:12.391875 18184 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44662 (request call id 722) took 1075 ms. Trace:
I20260807 03:39:12.392019 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:12.392271 18184 rpcz_store.cc:276] 0807 03:39:11.316763 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:11.317059 (+ 296us) service_pool.cc:227] Handling call
0807 03:39:12.391666 (+1074607us) spinlock_profiling.cc:243] Waited 140 ms on lock 0x7b54002cecd4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:12.391796 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":322757888}
I20260807 03:39:12.393775 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:12.394577 18190 raft_consensus.cc:1261] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260807 03:39:12.394966 18190 pending_rounds.cc:76] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02: Aborting all ops after (but not including) 1
W20260807 03:39:12.395056 20738 log.cc:924] Time spent T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 1.554s user 0.009s sys 0.000s
I20260807 03:39:12.395205 18190 pending_rounds.cc:97] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260807 03:39:12.395480 18190 raft_consensus.cc:2866] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 03:39:12.396003 20678 raft_consensus.cc:3032] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 03:39:12.396440 20678 raft_consensus.cc:729] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:12.397306 20678 consensus_queue.cc:260] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:12.363792 20829 raft_consensus.cc:2726] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057) with higher term. Message: Invalid argument: T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 5. Current term is 6.
I20260807 03:39:12.328466 20816 raft_consensus.cc:504] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:12.438982 20583 raft_consensus.cc:2726] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260807 03:39:12.449332 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 117707us
W20260807 03:39:12.430939 20753 log.cc:924] Time spent T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 1.097s user 0.000s sys 0.004s
W20260807 03:39:12.434257 20801 log.cc:924] Time spent T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.217s user 0.000s sys 0.007s
I20260807 03:39:12.469553 18027 leader_election.cc:302] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:39:12.470050 20816 leader_election.cc:288] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:12.470477 20829 raft_consensus.cc:2726] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 03:39:12.470701 18300 raft_consensus.cc:2454] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 5.
I20260807 03:39:12.471894 18027 leader_election.cc:302] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:12.434415 20803 raft_consensus.cc:2781] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:12.473294 20803 raft_consensus.cc:482] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:12.473362 20847 raft_consensus.cc:2781] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 03:39:12.473763 20803 raft_consensus.cc:3037] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:12.473995 20847 raft_consensus.cc:482] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:12.474467 20847 raft_consensus.cc:3037] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:12.476810 18027 leader_election.cc:398] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Vote denied by peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057) with higher term. Message: Invalid argument: T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 4. Current term is 5.
I20260807 03:39:12.477048 18310 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57268 (request call id 1045) took 1597 ms. Trace:
I20260807 03:39:12.477717 18027 leader_election.cc:302] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:12.477404 18310 rpcz_store.cc:276] 0807 03:39:10.879412 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:10.879695 (+ 283us) service_pool.cc:227] Handling call
0807 03:39:10.914749 (+ 35054us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:10.919641 (+ 4892us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:10.919684 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:39:10.962214 (+ 42530us) log.cc:842] Serialized 26 byte log entry
0807 03:39:11.231377 (+269163us) raft_consensus.cc:1631] Marking committed up to 2
0807 03:39:11.231425 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:11.231494 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:12.398952 (+1167458us) spinlock_profiling.cc:243] Waited 6.17 ms on lock 0x7b44000232cc. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f1a9a95 00007fe98f1aa4b2 00007fe991e52511 00007fe991e4de1a 00007fe991e41bee 00007fe991e40f36 00007fe991e40b58 00007fe991e4a551 00007fe991e4a509 00007fe991e4a499
0807 03:39:12.476764 (+ 77812us) raft_consensus.cc:1671] finished
0807 03:39:12.476783 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:12.476990 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":36525,"raft.queue_time_us":17003,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":23264,"spinlock_wait_cycles":14183808,"thread_start_us":223944,"threads_started":1,"wal-append.queue_time_us":330841}
I20260807 03:39:12.480284 20829 raft_consensus.cc:2726] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:12.481510 18027 leader_election.cc:302] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:12.485139 20124 raft_consensus.cc:504] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:12.486565 20556 raft_consensus.cc:482] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:12.488190 18185 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
mode: GRACEFUL
new_leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:34758
I20260807 03:39:12.488349 20556 raft_consensus.cc:504] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:12.488922 18185 raft_consensus.cc:595] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Received request to transfer leadership to TS d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:12.489269 18185 raft_consensus.cc:603] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 03:39:12.490386 20556 leader_election.cc:288] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:12.491734 20124 leader_election.cc:288] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:12.491863 18185 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4ba8413916634e039111c838fbe1e587"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
mode: GRACEFUL
new_leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:34758
I20260807 03:39:12.492448 18185 raft_consensus.cc:595] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Received request to transfer leadership to TS d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:12.492985 20829 raft_consensus.cc:2726] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260807 03:39:12.493904 20775 log.cc:924] Time spent T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.768s user 0.004s sys 0.000s
I20260807 03:39:12.494210 18027 leader_election.cc:302] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:12.494961 20829 raft_consensus.cc:2726] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:12.495303 18293 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57268 (request call id 1062) took 1033 ms. Trace:
I20260807 03:39:12.495683 18293 rpcz_store.cc:276] 0807 03:39:11.461456 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:11.461587 (+ 131us) service_pool.cc:227] Handling call
0807 03:39:11.461907 (+ 320us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 03:39:11.462246 (+ 339us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:11.462273 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 03:39:11.462774 (+ 501us) log.cc:842] Serialized 52 byte log entry
0807 03:39:11.630916 (+168142us) raft_consensus.cc:1631] Marking committed up to 3
0807 03:39:11.630952 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:11.631011 (+ 59us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:12.493276 (+862265us) spinlock_profiling.cc:243] Waited 2.55 ms on lock 0x7b44000232cc. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f1a9a95 00007fe98f1aa4b2 00007fe991e52511 00007fe991e4de1a 00007fe991e41bee 00007fe991e40f36 00007fe991e40b58 00007fe991e4a551 00007fe991e4a509 00007fe991e4a499
0807 03:39:12.494999 (+ 1723us) raft_consensus.cc:1671] finished
0807 03:39:12.495020 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:12.495242 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1558,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":109,"spinlock_wait_cycles":5857024,"thread_start_us":96324,"threads_started":1,"wal-append.queue_time_us":211707}
I20260807 03:39:12.486404 20847 raft_consensus.cc:504] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:12.486690 20620 raft_consensus.cc:3032] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:39:12.498880 18185 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
mode: GRACEFUL
new_leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:34758
I20260807 03:39:12.500728 18185 raft_consensus.cc:595] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Received request to transfer leadership to TS d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:12.501484 18185 raft_consensus.cc:603] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 03:39:12.462929 20822 consensus_queue.cc:1035] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.055s
I20260807 03:39:12.521304 18027 leader_election.cc:302] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:12.499927 20620 raft_consensus.cc:729] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
W20260807 03:39:12.528599 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 63112.8us
I20260807 03:39:12.539297 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:12.540478 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:12.542474 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:12.543038 18298 raft_consensus.cc:2454] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 4.
I20260807 03:39:12.540234 18190 raft_consensus.cc:2454] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 4.
W20260807 03:39:12.517345 20828 log.cc:924] Time spent T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
W20260807 03:39:12.516333 20820 log.cc:924] Time spent T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.088s user 0.001s sys 0.000s
I20260807 03:39:12.548696 18305 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57276 (request call id 786) took 1055 ms. Trace:
I20260807 03:39:12.552057 18185 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "60437eed8852484c88e84993e376670e"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
mode: GRACEFUL
new_leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:34758
I20260807 03:39:12.550172 18189 raft_consensus.cc:1226] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 LEADER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 1. Current term is 2. Ops: []
I20260807 03:39:12.553009 18185 raft_consensus.cc:595] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Received request to transfer leadership to TS d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:12.553500 18185 raft_consensus.cc:603] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 03:39:12.552234 18305 rpcz_store.cc:276] 0807 03:39:11.493045 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:11.688895 (+195850us) service_pool.cc:227] Handling call
0807 03:39:11.712980 (+ 24085us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:11.733716 (+ 20736us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:11.733747 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:39:11.762066 (+ 28319us) log.cc:842] Serialized 26 byte log entry
0807 03:39:11.890122 (+128056us) raft_consensus.cc:1631] Marking committed up to 2
0807 03:39:11.890215 (+ 93us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:11.890301 (+ 86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:12.548501 (+658200us) raft_consensus.cc:1671] finished
0807 03:39:12.548526 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:12.548638 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20554,"raft.queue_time_us":37967,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":1540,"thread_start_us":106125,"threads_started":1,"wal-append.queue_time_us":180896}
I20260807 03:39:12.556762 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:12.520510 20623 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.689s user 0.012s sys 0.028s
I20260807 03:39:12.558636 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:12.517648 20813 leader_election.cc:288] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:12.504531 19743 consensus_queue.cc:1035] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.043s
I20260807 03:39:12.562274 20620 consensus_queue.cc:260] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:12.562820 20847 leader_election.cc:288] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:12.548509 20829 consensus_queue.cc:1035] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
I20260807 03:39:12.534638 20798 raft_consensus.cc:2781] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260807 03:39:12.558109 20623 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.727s user 0.012s sys 0.028s
W20260807 03:39:12.571467 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 53900.5us
I20260807 03:39:12.517853 20682 consensus_queue.cc:1035] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20260807 03:39:12.518683 20277 consensus_queue.cc:1046] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.046s
I20260807 03:39:12.577298 19240 leader_election.cc:302] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:12.578310 18191 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
mode: GRACEFUL
new_leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:34758
I20260807 03:39:12.584542 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:12.584936 18191 raft_consensus.cc:595] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 7 LEADER]: Received request to transfer leadership to TS d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:12.511822 20839 consensus_queue.cc:1046] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.055s
I20260807 03:39:12.578658 20586 raft_consensus.cc:482] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:12.585258 18075 raft_consensus.cc:2454] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
I20260807 03:39:12.581508 18140 leader_election.cc:302] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:12.569509 20713 raft_consensus.cc:482] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:12.589125 20779 raft_consensus.cc:2726] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:12.575026 20798 raft_consensus.cc:482] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:12.567508 20803 raft_consensus.cc:504] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:12.602026 18027 leader_election.cc:302] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:12.602360 18070 raft_consensus.cc:1226] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 7. Current term is 8. Ops: []
I20260807 03:39:12.604825 18075 raft_consensus.cc:1261] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:39:12.606442 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:12.596299 20397 raft_consensus.cc:2781] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 03:39:12.612536 20752 raft_consensus.cc:482] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:12.613029 20752 raft_consensus.cc:504] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:12.613293 20722 consensus_queue.cc:1035] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:39:12.613484 20803 leader_election.cc:288] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:12.615093 20752 leader_election.cc:288] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:12.615700 20829 raft_consensus.cc:2781] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:12.603245 20798 raft_consensus.cc:3037] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:12.595826 20586 raft_consensus.cc:504] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:12.616068 20829 raft_consensus.cc:482] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:12.596557 20713 raft_consensus.cc:504] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:12.633559 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:12.634711 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:12.635253 18071 raft_consensus.cc:2396] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 03:39:12.635784 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:12.633891 20397 raft_consensus.cc:686] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:12.602064 20791 consensus_queue.cc:1046] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20260807 03:39:12.641968 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:12.644909 18070 raft_consensus.cc:2454] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:39:12.632022 20829 raft_consensus.cc:3037] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:12.651525 20840 consensus_queue.cc:1035] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20260807 03:39:12.654114 20713 leader_election.cc:288] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:12.657361 18071 raft_consensus.cc:2454] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:39:12.658244 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:12.660053 20397 consensus_queue.cc:237] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:12.637463 20722 consensus_queue.cc:1035] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
W20260807 03:39:12.842324 20865 log.cc:924] Time spent T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.211s user 0.001s sys 0.000s
I20260807 03:39:12.844781 18249 leader_election.cc:302] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
W20260807 03:39:12.845750 20849 log.cc:924] Time spent T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.074s user 0.000s sys 0.001s
I20260807 03:39:12.846920 20854 raft_consensus.cc:3032] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:39:12.847419 20854 raft_consensus.cc:729] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:12.848503 20854 consensus_queue.cc:260] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:12.850299 20854 raft_consensus.cc:3037] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:12.811576 20746 raft_consensus.cc:482] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 03:39:13.095830 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 251152us
W20260807 03:39:12.941280 20810 log.cc:924] Time spent T 00000000000000000000000000000000 P 8da2065a77024f35bdfc0dc3adb83f69: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20260807 03:39:13.103193 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:13.101042 20446 raft_consensus.cc:2726] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:13.092553 20746 raft_consensus.cc:504] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.142765 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:13.144255 20682 raft_consensus.cc:482] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:13.144596 20586 leader_election.cc:288] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:13.170231 18071 raft_consensus.cc:1261] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260807 03:39:13.212404 20746 leader_election.cc:288] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:13.215651 20874 log.cc:924] Time spent T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
W20260807 03:39:13.219585 18027 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.617s user 0.027s sys 0.042s
W20260807 03:39:13.219936 18027 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.617s user 0.027s sys 0.042s
W20260807 03:39:13.220237 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 618234us
I20260807 03:39:13.175551 20722 raft_consensus.cc:482] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:13.170688 20857 raft_consensus.cc:482] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:13.163743 20813 raft_consensus.cc:482] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 03:39:13.220022 20829 consensus_meta.cc:287] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 0.565s user 0.008s sys 0.000s
I20260807 03:39:13.228773 20556 raft_consensus.cc:482] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:13.255820 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:13.258028 18184 raft_consensus.cc:1226] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 4. Current term is 5. Ops: []
I20260807 03:39:13.259418 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:13.260002 18184 raft_consensus.cc:2454] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
I20260807 03:39:13.234762 20857 raft_consensus.cc:504] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.261735 18184 raft_consensus.cc:1261] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:39:13.262727 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:13.245474 20829 raft_consensus.cc:504] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.264503 18185 raft_consensus.cc:1226] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:13.265558 20829 leader_election.cc:288] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:13.245190 20803 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.522s user 0.005s sys 0.008s
I20260807 03:39:13.220803 20682 raft_consensus.cc:504] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.237812 20813 raft_consensus.cc:504] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:13.260005 20798 consensus_meta.cc:287] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 0.630s user 0.007s sys 0.004s
W20260807 03:39:13.281204 20886 log.cc:924] Time spent T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.060s user 0.001s sys 0.000s
I20260807 03:39:13.282244 20761 raft_consensus.cc:482] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:13.291965 18295 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37190 (request call id 24) took 2411 ms. Trace:
I20260807 03:39:13.232358 20722 raft_consensus.cc:504] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.296792 18295 rpcz_store.cc:276] 0807 03:39:10.880645 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:10.880820 (+ 175us) service_pool.cc:227] Handling call
0807 03:39:10.916000 (+ 35180us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:10.992194 (+ 76194us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:10.992223 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:39:10.992637 (+ 414us) log.cc:842] Serialized 26 byte log entry
0807 03:39:11.117944 (+125307us) raft_consensus.cc:1631] Marking committed up to 5
0807 03:39:13.148419 (+2030475us) spinlock_profiling.cc:243] Waited 1.4 ms on lock 0x7b18000b16a0. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe9981368b1 00007fe998143661 00007fe991e4e82c 00007fe98f2799dc 00007fe9980b0a39 00007fe98f2968bb 00007fe98f294cbf 00007fe98f294155 00007fe995b94167 00007fe9921c9807 00007fe9921d8f0f
0807 03:39:13.162894 (+ 14475us) log.cc:842] Serialized 14 byte log entry
0807 03:39:13.289643 (+126749us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:13.289731 (+ 88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:13.289797 (+ 66us) raft_consensus.cc:1671] finished
0807 03:39:13.291785 (+ 1988us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:13.291910 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":84539,"prepare.queue_time_us":170663,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":33,"raft.queue_time_us":86032,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":12743,"spinlock_wait_cycles":3220352,"thread_start_us":360026,"threads_started":3,"wal-append.queue_time_us":248456,"wal-append.run_cpu_time_us":7696,"wal-append.run_wall_time_us":1014125}
I20260807 03:39:13.262183 20556 raft_consensus.cc:504] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.300279 20764 raft_consensus.cc:482] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5c9b5062bdfa46af9396f1a85025a331)
I20260807 03:39:13.306746 20840 raft_consensus.cc:482] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:13.317305 20900 raft_consensus.cc:482] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:13.317420 20840 raft_consensus.cc:504] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.317922 20900 raft_consensus.cc:504] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.318825 20556 leader_election.cc:288] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:13.319664 20840 leader_election.cc:288] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:13.310840 20893 raft_consensus.cc:482] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:13.320386 20893 raft_consensus.cc:504] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.285079 20798 raft_consensus.cc:504] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.306046 20761 raft_consensus.cc:504] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.329093 20888 raft_consensus.cc:482] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:13.329610 20888 raft_consensus.cc:504] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.330657 20761 leader_election.cc:288] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:13.316408 20764 raft_consensus.cc:504] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:13.335685 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:13.325001 20840 raft_consensus.cc:482] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:13.336421 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:13.336913 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:13.337117 20840 raft_consensus.cc:504] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.337637 18310 raft_consensus.cc:2396] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 03:39:13.339061 20840 leader_election.cc:288] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:13.340647 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:13.318337 20722 leader_election.cc:288] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:13.354197 18295 raft_consensus.cc:1261] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260807 03:39:13.323258 20397 raft_consensus.cc:482] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:13.357394 20902 raft_consensus.cc:482] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:13.357772 20900 leader_election.cc:288] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:13.363106 20893 leader_election.cc:288] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:13.368614 20902 raft_consensus.cc:504] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:13.374503 20734 log.cc:924] Time spent T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 2.492s user 0.012s sys 0.012s
I20260807 03:39:13.377584 20798 leader_election.cc:288] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:13.378686 20682 leader_election.cc:288] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:13.379624 20857 leader_election.cc:288] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:13.356563 20889 raft_consensus.cc:482] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5c9b5062bdfa46af9396f1a85025a331)
I20260807 03:39:13.380733 20813 leader_election.cc:288] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:13.381409 20397 raft_consensus.cc:504] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.391848 20761 raft_consensus.cc:482] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:13.400708 20688 raft_consensus.cc:482] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:13.418387 20910 raft_consensus.cc:482] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:13.423539 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:13.424499 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:13.424715 18305 raft_consensus.cc:3032] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:39:13.445371 18305 raft_consensus.cc:729] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:13.425381 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:13.427279 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:13.427776 20902 leader_election.cc:288] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:13.427903 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:13.428190 20613 raft_consensus.cc:482] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:13.429775 20861 raft_consensus.cc:482] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:13.430215 20042 raft_consensus.cc:482] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
W20260807 03:39:13.430721 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 209850us
I20260807 03:39:13.431475 20694 raft_consensus.cc:3037] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:13.431945 20853 raft_consensus.cc:482] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:13.432438 20880 raft_consensus.cc:482] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:13.429805 20804 raft_consensus.cc:482] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:13.419106 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:13.420601 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:13.419833 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:13.447535 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:13.449110 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:13.451854 18249 leader_election.cc:302] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:13.445201 20803 raft_consensus.cc:482] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:13.455770 20613 raft_consensus.cc:504] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:13.457774 18003 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 4a8b888de6f046a88c6832c234e5de79, TS 66a36e1ac8e945e893668e282d394c49 -> TS 5c9b5062bdfa46af9396f1a85025a331 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:13.530041 18003 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 4a8b888de6f046a88c6832c234e5de79, TS 66a36e1ac8e945e893668e282d394c49 -> TS 5c9b5062bdfa46af9396f1a85025a331 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.530721 18300 raft_consensus.cc:2379] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 2: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:13.446550 18305 consensus_queue.cc:260] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.531383 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:13.412477 20893 raft_consensus.cc:482] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:13.456271 20703 raft_consensus.cc:482] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:13.511644 18027 leader_election.cc:302] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:13.462767 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:13.439213 20653 raft_consensus.cc:3032] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:39:13.517606 20909 consensus_queue.cc:1035] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20260807 03:39:13.517994 20902 raft_consensus.cc:482] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:13.418738 20889 raft_consensus.cc:504] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:13.521483 20888 leader_election.cc:288] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:13.450737 20910 raft_consensus.cc:504] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.454094 20658 raft_consensus.cc:482] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:13.446002 20688 raft_consensus.cc:504] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:13.528911 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 77132.2us
I20260807 03:39:13.534966 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:13.527532 18193 raft_consensus.cc:2454] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:39:13.581511 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:13.630793 18298 raft_consensus.cc:3037] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:13.428217 20761 raft_consensus.cc:504] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.461128 20857 raft_consensus.cc:482] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:13.617624 18072 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33254 (request call id 1204) took 1034 ms. Trace:
I20260807 03:39:13.622177 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:13.622184 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:13.524036 20764 leader_election.cc:288] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:13.624199 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:13.532922 20682 raft_consensus.cc:2726] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 03:39:13.617031 20498 raft_consensus.cc:2781] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:39:13.619673 20813 raft_consensus.cc:482] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:13.510123 20861 raft_consensus.cc:504] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.600472 20795 raft_consensus.cc:482] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:13.596990 20653 raft_consensus.cc:729] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:13.643078 18072 rpcz_store.cc:276] 0807 03:39:12.583461 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:12.583568 (+ 107us) service_pool.cc:227] Handling call
0807 03:39:12.593774 (+ 10206us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:12.596405 (+ 2631us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:12.596449 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:39:12.596907 (+ 458us) log.cc:842] Serialized 26 byte log entry
0807 03:39:13.378128 (+781221us) spinlock_profiling.cc:243] Waited 5.38 ms on lock 0x7b18003abe80. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe9981368b1 00007fe998143661 00007fe98f279345 00007fe9980b0a39 00007fe98f2968bb 00007fe98f294cbf 00007fe98f296e22 00007fe991e405bb 00007fe991e46063 00007fe991e46411 00007fe9921935cf
0807 03:39:13.394976 (+ 16848us) raft_consensus.cc:1631] Marking committed up to 2
0807 03:39:13.395020 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:13.395102 (+ 82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:13.524159 (+129057us) raft_consensus.cc:1671] finished
0807 03:39:13.524182 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:13.617536 (+ 93354us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2240,"raft.queue_time_us":10485,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":5359,"spinlock_wait_cycles":12384000,"thread_start_us":16651,"threads_started":1,"wal-append.queue_time_us":845234}
W20260807 03:39:13.700729 18003 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet b2d369700aef4621b22ba058956d0894, TS d9716e2bf06442f5b3b0e81087c53c4d -> TS 5c9b5062bdfa46af9396f1a85025a331 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:13.703126 18003 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet b2d369700aef4621b22ba058956d0894, TS d9716e2bf06442f5b3b0e81087c53c4d -> TS 5c9b5062bdfa46af9396f1a85025a331 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.621526 20713 raft_consensus.cc:482] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:13.618188 20586 raft_consensus.cc:482] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:13.656982 18249 leader_election.cc:302] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
W20260807 03:39:13.709159 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 52307us
I20260807 03:39:13.651891 18193 raft_consensus.cc:1261] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 03:39:13.513231 20042 raft_consensus.cc:504] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:13.624136 20847 raft_consensus.cc:482] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:13.717665 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:13.718474 18193 raft_consensus.cc:3037] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:13.720376 20862 raft_consensus.cc:482] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:13.720922 20862 raft_consensus.cc:504] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.722957 20862 leader_election.cc:288] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:13.667176 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
W20260807 03:39:13.669449 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 205511us
I20260807 03:39:13.625788 20902 raft_consensus.cc:504] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.669629 18300 raft_consensus.cc:1261] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 03:39:13.688105 18293 raft_consensus.cc:2396] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 03:39:13.688355 18027 leader_election.cc:302] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
W20260807 03:39:13.711179 20758 log.cc:924] Time spent T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 2.382s user 0.017s sys 0.004s
I20260807 03:39:13.731427 18312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57268 (request call id 1050) took 2580 ms. Trace:
I20260807 03:39:13.732236 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:13.733291 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:13.734287 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:13.734856 18180 raft_consensus.cc:2454] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3.
I20260807 03:39:13.734289 20798 raft_consensus.cc:482] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:13.735692 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:13.726289 20847 raft_consensus.cc:504] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.737950 20847 leader_election.cc:288] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:13.731690 20653 consensus_queue.cc:260] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.740940 20688 leader_election.cc:288] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:13.741227 18079 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33254 (request call id 1202) took 1158 ms. Trace:
I20260807 03:39:13.741743 18079 rpcz_store.cc:276] 0807 03:39:12.582971 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:12.583092 (+ 121us) service_pool.cc:227] Handling call
0807 03:39:12.583321 (+ 229us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:12.634392 (+ 51071us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:12.634434 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:12.634495 (+ 61us) raft_consensus.cc:1631] Marking committed up to 2
0807 03:39:12.634533 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:13.740920 (+1106387us) spinlock_profiling.cc:243] Waited 50.9 ms on lock 0x7b54003782d4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:39:13.740990 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:13.741160 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":116999424}
I20260807 03:39:13.742650 20613 leader_election.cc:288] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:13.693405 20795 raft_consensus.cc:504] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.717170 20586 raft_consensus.cc:504] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.745940 20586 leader_election.cc:288] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:13.598714 20803 raft_consensus.cc:504] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.747977 20803 leader_election.cc:288] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:13.637964 20888 raft_consensus.cc:482] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:13.749490 20888 raft_consensus.cc:504] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.751430 20888 leader_election.cc:288] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:13.650092 20913 raft_consensus.cc:482] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:13.752472 20913 raft_consensus.cc:504] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.754540 20913 leader_election.cc:288] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:13.756139 20761 leader_election.cc:288] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:13.757853 20861 leader_election.cc:288] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:13.758543 20928 raft_consensus.cc:482] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:13.759089 20928 raft_consensus.cc:504] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.761662 20928 leader_election.cc:288] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:13.731904 18312 rpcz_store.cc:276] 0807 03:39:11.151359 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:11.151533 (+ 174us) service_pool.cc:227] Handling call
0807 03:39:11.161154 (+ 9621us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:11.179348 (+ 18194us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:11.179377 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:39:11.179826 (+ 449us) log.cc:842] Serialized 26 byte log entry
0807 03:39:11.306720 (+126894us) raft_consensus.cc:1631] Marking committed up to 1
0807 03:39:11.306756 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:11.306824 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:13.628023 (+2321199us) spinlock_profiling.cc:243] Waited 6.7 ms on lock 0x7b44000232cc. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f1a9a95 00007fe98f1aa4b2 00007fe991e52511 00007fe991e4de1a 00007fe991e41bee 00007fe991e40f36 00007fe991e40b58 00007fe991e4a551 00007fe991e4a509 00007fe991e4a499
0807 03:39:13.731118 (+103095us) raft_consensus.cc:1671] finished
0807 03:39:13.731139 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:13.731359 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17839,"raft.queue_time_us":18121,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":135,"spinlock_wait_cycles":15414144,"thread_start_us":126565,"threads_started":1,"wal-append.queue_time_us":148525}
I20260807 03:39:13.765228 20397 leader_election.cc:288] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:13.766359 20713 raft_consensus.cc:504] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.768509 20713 leader_election.cc:288] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:13.770136 20795 leader_election.cc:288] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:13.770931 20397 raft_consensus.cc:482] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:13.771413 20397 raft_consensus.cc:504] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.774217 20397 leader_election.cc:288] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:13.775993 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 48897.2us
I20260807 03:39:13.778034 18249 leader_election.cc:302] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:13.779142 19743 raft_consensus.cc:2781] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:13.779501 19743 raft_consensus.cc:482] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:13.729418 20812 consensus_queue.cc:1035] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.051s
I20260807 03:39:13.780575 20677 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:13.781100 20677 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.654053 20498 raft_consensus.cc:482] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:13.782222 20498 raft_consensus.cc:3037] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:13.783228 20677 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:13.784207 18180 raft_consensus.cc:2379] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 3: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:13.785307 20397 raft_consensus.cc:482] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 03:39:13.785624 20920 log.cc:924] Time spent T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.100s user 0.000s sys 0.000s
I20260807 03:39:13.687294 20857 raft_consensus.cc:504] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.779888 19743 raft_consensus.cc:3037] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:13.787433 20857 leader_election.cc:288] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:13.787848 20902 leader_election.cc:288] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:13.779778 20761 consensus_queue.cc:1035] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.009s
I20260807 03:39:13.789726 20854 raft_consensus.cc:482] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:13.779162 20399 consensus_queue.cc:1046] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20260807 03:39:13.790234 20854 raft_consensus.cc:504] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.616717 20703 raft_consensus.cc:504] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.785794 20397 raft_consensus.cc:504] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.795819 19743 raft_consensus.cc:504] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.533612 20804 raft_consensus.cc:504] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.598348 20893 raft_consensus.cc:504] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.528981 20880 raft_consensus.cc:504] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.703606 19297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:13.695402 20651 raft_consensus.cc:482] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:13.667133 20682 raft_consensus.cc:482] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:13.528419 20853 raft_consensus.cc:504] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.692690 20813 raft_consensus.cc:504] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:13.735793 20881 log.cc:924] Time spent T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.610s user 0.005s sys 0.003s
I20260807 03:39:13.735538 20798 raft_consensus.cc:504] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.797139 20397 leader_election.cc:288] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:13.797821 20928 raft_consensus.cc:482] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:13.659667 20583 raft_consensus.cc:482] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:13.869144 20862 raft_consensus.cc:482] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:13.869324 20583 raft_consensus.cc:504] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.869758 20862 raft_consensus.cc:504] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.871582 20583 leader_election.cc:288] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:13.872273 20804 leader_election.cc:288] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:13.873121 20928 raft_consensus.cc:504] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.875511 20928 leader_election.cc:288] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:13.878687 20173 raft_consensus.cc:3032] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 LEADER]: Stepping down as leader of term 4
I20260807 03:39:13.879189 20173 raft_consensus.cc:729] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:13.650172 20658 raft_consensus.cc:504] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.741264 20042 leader_election.cc:288] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:13.806160 20703 leader_election.cc:288] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:13.691208 18298 raft_consensus.cc:2454] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 5.
W20260807 03:39:13.838128 18003 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 303e3bbfb1c74e1686eae36683a96314, TS 66a36e1ac8e945e893668e282d394c49 -> TS 5c9b5062bdfa46af9396f1a85025a331 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:13.909472 18003 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 303e3bbfb1c74e1686eae36683a96314, TS 66a36e1ac8e945e893668e282d394c49 -> TS 5c9b5062bdfa46af9396f1a85025a331 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.724752 20927 raft_consensus.cc:482] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
W20260807 03:39:13.866933 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 178645us
W20260807 03:39:13.867086 20918 log.cc:924] Time spent T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.241s user 0.002s sys 0.000s
I20260807 03:39:13.836222 20902 raft_consensus.cc:482] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:13.939340 20902 raft_consensus.cc:504] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.940861 20880 leader_election.cc:288] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:13.896003 20173 consensus_queue.cc:260] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.905435 20857 raft_consensus.cc:482] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:13.906901 20910 leader_election.cc:288] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:13.854583 20682 raft_consensus.cc:504] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.944572 20857 raft_consensus.cc:504] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.868244 20691 raft_consensus.cc:2726] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:13.841650 20651 raft_consensus.cc:504] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.951913 20927 raft_consensus.cc:504] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.881476 20583 raft_consensus.cc:482] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:13.917641 20446 raft_consensus.cc:2741] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260807 03:39:13.984764 20853 leader_election.cc:288] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:13.985960 20691 raft_consensus.cc:482] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:13.986467 20691 raft_consensus.cc:504] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.987350 20498 raft_consensus.cc:504] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.988433 20862 leader_election.cc:288] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:13.989493 20498 leader_election.cc:288] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:13.990165 19292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:13.990816 19743 leader_election.cc:288] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:13.993674 18190 raft_consensus.cc:1226] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 4. Current term is 5. Ops: []
I20260807 03:39:13.993505 20583 raft_consensus.cc:504] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.994902 18190 raft_consensus.cc:1226] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 4. Current term is 5. Ops: []
I20260807 03:39:13.983278 20718 consensus_queue.cc:260] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:13.995581 20718 raft_consensus.cc:3037] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:13.997326 20583 leader_election.cc:288] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:13.998242 20854 leader_election.cc:288] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:13.988517 20691 leader_election.cc:288] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:14.027299 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.028061 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.028877 18312 raft_consensus.cc:2454] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 5.
I20260807 03:39:14.001767 18184 raft_consensus.cc:1226] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 2. Current term is 3. Ops: []
I20260807 03:39:14.004897 20857 leader_election.cc:288] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
W20260807 03:39:14.014940 20906 log.cc:924] Time spent T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.698s user 0.003s sys 0.000s
I20260807 03:39:14.015865 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.023674 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.025390 18073 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33254 (request call id 1205) took 1441 ms. Trace:
I20260807 03:39:14.031613 18184 raft_consensus.cc:1226] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 4. Current term is 5. Ops: []
I20260807 03:39:14.037752 18073 rpcz_store.cc:276] 0807 03:39:12.583656 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:12.583783 (+ 127us) service_pool.cc:227] Handling call
0807 03:39:12.591684 (+ 7901us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 03:39:12.592035 (+ 351us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:12.592065 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 03:39:12.592683 (+ 618us) log.cc:842] Serialized 52 byte log entry
0807 03:39:12.631626 (+ 38943us) raft_consensus.cc:1631] Marking committed up to 2
0807 03:39:12.639256 (+ 7630us) log.cc:842] Serialized 14 byte log entry
0807 03:39:13.423358 (+784102us) spinlock_profiling.cc:243] Waited 4.07 ms on lock 0x7b18003aba60. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe9981368b1 00007fe998143661 00007fe98f20e4ca 00007fe9988eff99 00007fe98f0d2990 00007fe98f294864 00007fe98f294155 00007fe9921b73d0 00007fe9921c9576 00007fe9921da0d7 00007fe9921da058
0807 03:39:13.423527 (+ 169us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:13.423591 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:13.798488 (+374897us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b18003aba60. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe9981368b1 00007fe998143661 00007fe98f297638 00007fe98f29ab11 00007fe98f29aac9 00007fe98f29aa59 00007fe98f29aa21 00007fe98f299d1d 00007fe9984fb204 00007fe9984fb039 00007fe98f279dc6
0807 03:39:14.025148 (+226660us) raft_consensus.cc:1671] finished
0807 03:39:14.025177 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:14.025326 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2491,"prepare.queue_time_us":7456,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":56,"raft.queue_time_us":790468,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":5533,"spinlock_wait_cycles":35560448,"thread_start_us":38508,"threads_started":1,"wal-append.queue_time_us":66806}
I20260807 03:39:13.983902 20682 leader_election.cc:288] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:14.032022 18193 raft_consensus.cc:2454] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 5.
I20260807 03:39:14.007941 20798 leader_election.cc:288] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:14.025979 20173 raft_consensus.cc:3037] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:14.173398 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.174443 18300 raft_consensus.cc:2379] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 4: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:14.183079 20902 leader_election.cc:288] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:14.165287 20857 raft_consensus.cc:482] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:14.170576 20691 raft_consensus.cc:482] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:14.200991 20691 raft_consensus.cc:504] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:14.165596 20854 raft_consensus.cc:482] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:14.204943 20854 raft_consensus.cc:504] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:14.207530 20854 leader_election.cc:288] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:14.209851 18293 raft_consensus.cc:1226] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:14.169219 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.169064 20583 raft_consensus.cc:482] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:14.169263 20804 raft_consensus.cc:482] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:14.213251 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.203859 20691 leader_election.cc:288] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:14.248966 18141 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.520s user 0.111s sys 0.139s
W20260807 03:39:14.249341 18141 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.520s user 0.111s sys 0.139s
W20260807 03:39:14.249631 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 520862us
I20260807 03:39:14.225345 20857 raft_consensus.cc:504] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:14.255093 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.211160 20880 raft_consensus.cc:482] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:14.222651 20862 raft_consensus.cc:482] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:14.224591 20853 raft_consensus.cc:482] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:14.261211 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.250181 20813 leader_election.cc:288] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:14.226965 20804 raft_consensus.cc:504] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:14.267144 20804 leader_election.cc:288] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:14.268653 20853 raft_consensus.cc:504] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:14.270570 20853 leader_election.cc:288] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:14.274829 20889 leader_election.cc:288] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:14.275454 20893 leader_election.cc:288] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:14.277063 19292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:14.280858 20893 consensus_queue.cc:1035] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:39:14.281267 18188 raft_consensus.cc:3037] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:14.261641 20880 raft_consensus.cc:504] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:14.272109 20862 raft_consensus.cc:504] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:14.297387 20862 leader_election.cc:288] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:14.303339 20880 leader_election.cc:288] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:14.307717 18081 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33268 (request call id 989) took 3362 ms. Trace:
I20260807 03:39:14.308184 18081 rpcz_store.cc:276] 0807 03:39:10.945474 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:10.945588 (+ 114us) service_pool.cc:227] Handling call
0807 03:39:12.109069 (+1163481us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:12.109572 (+ 503us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:14.307441 (+2197869us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b5400341a50. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bea09 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:14.307615 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2632058112}
I20260807 03:39:14.311858 18293 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8c98c1d422654739af06653a512f7499"
dest_uuid: "66a36e1ac8e945e893668e282d394c49"
mode: GRACEFUL
new_leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:58736
I20260807 03:39:14.312724 20620 raft_consensus.cc:3037] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:14.225039 20583 raft_consensus.cc:504] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:14.315328 20583 leader_election.cc:288] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:14.319782 20816 raft_consensus.cc:3032] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 03:39:14.320279 20816 raft_consensus.cc:729] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:14.321267 20816 consensus_queue.cc:260] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:14.322590 20816 raft_consensus.cc:3037] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:14.324209 18189 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44660 (request call id 1271) took 2063 ms. Trace:
I20260807 03:39:14.312738 18293 raft_consensus.cc:595] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Received request to transfer leadership to TS d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:14.324587 18189 rpcz_store.cc:276] 0807 03:39:12.261273 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:12.261427 (+ 154us) service_pool.cc:227] Handling call
0807 03:39:12.274507 (+ 13080us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:12.553340 (+278833us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:14.323943 (+1770603us) spinlock_profiling.cc:243] Waited 275 ms on lock 0x7b54001dea54. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:39:14.324151 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":633286528}
I20260807 03:39:14.326507 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
W20260807 03:39:14.338963 20916 log.cc:924] Time spent T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.613s user 0.001s sys 0.010s
I20260807 03:39:14.345465 20653 raft_consensus.cc:3037] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:14.359019 20651 leader_election.cc:288] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:14.367270 20882 raft_consensus.cc:482] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:14.370132 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:14.370718 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:14.372714 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:14.373308 18184 raft_consensus.cc:2454] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3.
I20260807 03:39:14.374177 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:14.373013 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:14.371493 18193 raft_consensus.cc:2454] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3.
I20260807 03:39:14.376209 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:14.376384 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:14.376520 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:14.376796 18185 raft_consensus.cc:2454] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3.
I20260807 03:39:14.377391 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:14.378042 18180 raft_consensus.cc:2379] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 4: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:14.378324 18188 raft_consensus.cc:1261] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260807 03:39:14.378731 18188 pending_rounds.cc:76] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02: Aborting all ops after (but not including) 2
I20260807 03:39:14.378978 18188 pending_rounds.cc:97] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260807 03:39:14.379242 18188 raft_consensus.cc:2866] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 03:39:14.386047 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:14.386260 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.386799 18293 raft_consensus.cc:2454] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 4.
I20260807 03:39:14.387660 18309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57276 (request call id 785) took 2894 ms. Trace:
I20260807 03:39:14.379921 20746 raft_consensus.cc:482] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:14.367805 20882 raft_consensus.cc:504] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:14.388765 20746 raft_consensus.cc:504] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:14.390547 20882 leader_election.cc:288] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:14.387743 18248 leader_election.cc:334] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): Timed out: RequestConsensusVote RPC to 127.15.198.129:36163 timed out after 1.777s (SENT)
I20260807 03:39:14.392481 18248 leader_election.cc:302] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:14.416463 18188 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44660 (request call id 1267) took 2695 ms. Trace:
I20260807 03:39:14.418072 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.388113 18309 rpcz_store.cc:276] 0807 03:39:11.492769 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:11.687462 (+194693us) service_pool.cc:227] Handling call
0807 03:39:11.726667 (+ 39205us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:11.827250 (+100583us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:11.827278 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:39:12.591115 (+763837us) spinlock_profiling.cc:243] Waited 6.56 ms on lock 0x7b2c00108448. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe990ba2e11 00007fe990bb2442 00007fe9921fac11 00007fe9921bf58a 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772
0807 03:39:12.591441 (+ 326us) log.cc:842] Serialized 26 byte log entry
0807 03:39:12.609591 (+ 18150us) raft_consensus.cc:1631] Marking committed up to 1
0807 03:39:12.609622 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:14.387347 (+1777725us) spinlock_profiling.cc:243] Waited 65.8 ms on lock 0x7b540018bad4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:39:14.387414 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:14.387461 (+ 47us) raft_consensus.cc:1671] finished
0807 03:39:14.387473 (+ 12us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:14.387609 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34307,"raft.queue_time_us":58538,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":99,"spinlock_wait_cycles":166516352,"thread_start_us":17823,"threads_started":1,"wal-append.queue_time_us":58756,"wal-append.run_cpu_time_us":1796,"wal-append.run_wall_time_us":1001852}
W20260807 03:39:14.374302 20765 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.797s user 0.042s sys 0.098s
I20260807 03:39:14.456473 20653 raft_consensus.cc:482] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 03:39:14.457217 20765 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.858s user 0.042s sys 0.098s
I20260807 03:39:14.431679 20620 raft_consensus.cc:2726] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 03:39:14.466627 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:14.467604 18193 raft_consensus.cc:2454] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 5.
I20260807 03:39:14.468989 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:14.469887 18180 raft_consensus.cc:2379] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 5: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:14.417002 18188 rpcz_store.cc:276] 0807 03:39:11.684888 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:11.691222 (+ 6334us) service_pool.cc:227] Handling call
0807 03:39:11.691475 (+ 253us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:14.280900 (+2589425us) spinlock_profiling.cc:243] Waited 2.21 s on lock 0x7b580028c4b0. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe99213af63 00007fe99219eaba 00007fe9921b63af 00007fe9921b7b4e 00007fe9921c1c89 00007fe9921c2556 00007fe9921beedf 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684
0807 03:39:14.379737 (+ 98837us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:14.379974 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5082050816}
I20260807 03:39:14.471006 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:14.472450 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:14.473750 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:39:14.473958 18188 raft_consensus.cc:2454] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 5.
W20260807 03:39:14.474690 18191 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.663s user 0.005s sys 0.009s
W20260807 03:39:14.475576 18191 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.664s user 0.006s sys 0.009s
I20260807 03:39:14.476557 20880 raft_consensus.cc:482] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:14.476722 20765 raft_consensus.cc:482] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:14.477098 20880 raft_consensus.cc:504] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:14.477288 20765 raft_consensus.cc:504] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:14.479156 20880 leader_election.cc:288] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:14.479841 18081 raft_consensus.cc:1261] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 03:39:14.480341 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:14.484519 18140 leader_election.cc:302] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:14.486012 18140 leader_election.cc:302] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:14.488607 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.488924 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.489957 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:14.490533 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.490713 18082 raft_consensus.cc:2454] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:39:14.491380 18293 raft_consensus.cc:2454] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 5.
I20260807 03:39:14.492542 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:14.492836 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:14.493247 18311 raft_consensus.cc:3037] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:14.494139 18072 raft_consensus.cc:1226] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:14.496176 18027 leader_election.cc:302] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:14.496759 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:14.497509 18072 raft_consensus.cc:2454] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 7.
I20260807 03:39:14.499373 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:14.501442 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.502029 18298 raft_consensus.cc:2454] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:39:14.502364 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:14.502727 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.503099 18069 raft_consensus.cc:2454] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 6.
I20260807 03:39:14.503751 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ca0223f47b46fd955704eba5259370" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.479630 20765 leader_election.cc:288] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:14.506058 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:14.508211 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.508431 18027 leader_election.cc:302] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:14.509240 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.510037 18298 raft_consensus.cc:2379] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 6: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:14.504153 18140 leader_election.cc:302] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:14.505478 18072 raft_consensus.cc:2379] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 2: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:14.510088 18027 leader_election.cc:302] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:14.511399 18140 leader_election.cc:302] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:14.512974 18140 leader_election.cc:302] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:14.513093 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.473057 20857 leader_election.cc:288] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:14.516901 18080 raft_consensus.cc:1261] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 03:39:14.519562 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.513723 20853 raft_consensus.cc:2781] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election won for term 5
I20260807 03:39:14.523828 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:14.524583 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.524613 18075 raft_consensus.cc:2454] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
I20260807 03:39:14.526113 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.527349 18298 raft_consensus.cc:2454] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:39:14.528247 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.528925 18027 leader_election.cc:302] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:14.523272 20938 raft_consensus.cc:2726] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 03:39:14.480861 20583 raft_consensus.cc:482] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:14.504559 20615 raft_consensus.cc:2726] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:14.457250 20653 raft_consensus.cc:504] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:14.505075 20912 raft_consensus.cc:2726] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:14.531255 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:14.548223 18081 raft_consensus.cc:2454] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:14.515992 20684 raft_consensus.cc:2781] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:39:14.513240 20546 raft_consensus.cc:2726] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 03:39:14.516289 20804 raft_consensus.cc:2726] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:14.527670 20765 raft_consensus.cc:2781] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:39:14.554263 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.561923 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:14.566769 20933 log.cc:924] Time spent T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20260807 03:39:14.568303 20123 raft_consensus.cc:3037] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:14.513995 20862 consensus_queue.cc:1035] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.043s
I20260807 03:39:14.539060 20853 raft_consensus.cc:686] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:14.574162 20853 consensus_queue.cc:237] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:14.541105 20880 raft_consensus.cc:2781] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:14.577085 20880 raft_consensus.cc:482] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:14.577577 20880 raft_consensus.cc:3037] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:14.578162 20927 leader_election.cc:288] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:14.585563 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:14.588999 20658 leader_election.cc:288] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:14.596783 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:14.599524 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 63682.5us
I20260807 03:39:14.531965 20694 consensus_queue.cc:1046] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
I20260807 03:39:14.605176 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.605940 18309 raft_consensus.cc:2454] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 5.
I20260807 03:39:14.615378 20653 leader_election.cc:288] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:14.597714 18075 raft_consensus.cc:2454] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
I20260807 03:39:14.618495 18311 raft_consensus.cc:2454] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 6.
I20260807 03:39:14.563457 20765 raft_consensus.cc:482] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:14.634469 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:14.637101 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:14.638468 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:14.639129 18184 raft_consensus.cc:2454] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:14.640048 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:14.643923 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:14.644789 18184 raft_consensus.cc:2379] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 5: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:14.545004 20583 raft_consensus.cc:504] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:14.654044 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:14.654206 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.654698 18309 raft_consensus.cc:2454] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:39:14.655128 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:14.655205 20583 leader_election.cc:288] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:14.655694 18189 raft_consensus.cc:2454] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:39:14.655929 18082 raft_consensus.cc:2454] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:14.656538 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:14.657657 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:14.658442 18075 raft_consensus.cc:2454] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 7.
I20260807 03:39:14.659632 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:14.592851 20684 raft_consensus.cc:686] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:14.653187 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:14.662693 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:14.663342 18185 raft_consensus.cc:2454] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:14.664294 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:14.664888 18185 raft_consensus.cc:3037] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:14.651011 20765 raft_consensus.cc:3037] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:14.667879 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:14.669034 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.651863 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.650369 20694 raft_consensus.cc:482] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:14.675256 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:14.676033 18081 raft_consensus.cc:2454] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:14.677862 18075 raft_consensus.cc:1261] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:39:14.686874 20173 raft_consensus.cc:482] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:14.689292 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.689580 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:14.693058 18027 leader_election.cc:302] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:14.696952 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:14.697712 18075 raft_consensus.cc:2454] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
I20260807 03:39:14.693608 18185 raft_consensus.cc:2454] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
I20260807 03:39:14.712550 18027 leader_election.cc:302] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:14.713933 18027 leader_election.cc:302] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:14.683147 20798 raft_consensus.cc:482] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:14.720619 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:14.720778 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 261285us
I20260807 03:39:14.719166 18027 leader_election.cc:302] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:14.723979 20543 consensus_queue.cc:1046] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 03:39:14.724956 20543 consensus_queue.cc:1225] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer 3c4bceb836db4536a0f255fdf86abc02 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260807 03:39:14.679220 20805 consensus_queue.cc:1046] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.032s
I20260807 03:39:14.727191 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.728024 18081 raft_consensus.cc:1261] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260807 03:39:14.696580 20939 raft_consensus.cc:482] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:14.679769 20694 raft_consensus.cc:504] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:14.730360 18309 raft_consensus.cc:1226] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260807 03:39:14.729476 18297 raft_consensus.cc:2454] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 5.
W20260807 03:39:14.738245 20941 log.cc:924] Time spent T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20260807 03:39:14.738449 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:14.738772 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:14.739868 18190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44662 (request call id 763) took 1045 ms. Trace:
I20260807 03:39:14.740301 20691 raft_consensus.cc:3037] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:14.741484 18070 raft_consensus.cc:1226] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:14.745100 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:14.745659 18080 raft_consensus.cc:3037] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:14.746887 20805 consensus_queue.cc:1225] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer 3c4bceb836db4536a0f255fdf86abc02 log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260807 03:39:14.747427 20805 raft_consensus.cc:3032] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 03:39:14.747704 20805 raft_consensus.cc:729] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:14.748584 18075 raft_consensus.cc:1226] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:14.748453 20805 consensus_queue.cc:260] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:14.749558 20805 raft_consensus.cc:3037] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:14.740319 18190 rpcz_store.cc:276] 0807 03:39:13.694502 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:14.022089 (+327587us) service_pool.cc:227] Handling call
0807 03:39:14.022447 (+ 358us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:14.022915 (+ 468us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:14.022951 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:39:14.023420 (+ 469us) log.cc:842] Serialized 26 byte log entry
0807 03:39:14.418407 (+394987us) raft_consensus.cc:1631] Marking committed up to 1
0807 03:39:14.418446 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:14.418518 (+ 72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:14.739592 (+321074us) raft_consensus.cc:1671] finished
0807 03:39:14.739633 (+ 41us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:14.739802 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":101,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":7619,"thread_start_us":394702,"threads_started":1,"wal-append.queue_time_us":647281}
I20260807 03:39:14.762748 18080 raft_consensus.cc:2454] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
I20260807 03:39:14.764345 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.728526 20804 raft_consensus.cc:2726] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 03:39:14.728214 20862 raft_consensus.cc:2726] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260807 03:39:14.732530 20583 raft_consensus.cc:2781] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:14.728574 20857 raft_consensus.cc:482] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:14.700949 20173 raft_consensus.cc:504] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:14.771413 20746 leader_election.cc:288] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:14.726440 20684 consensus_queue.cc:237] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:14.756548 20939 raft_consensus.cc:504] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:14.775853 18305 raft_consensus.cc:3037] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:14.733996 20722 raft_consensus.cc:3032] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:39:14.728633 20798 raft_consensus.cc:504] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:14.777590 20800 raft_consensus.cc:482] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:14.794044 20583 raft_consensus.cc:482] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:14.816609 20583 raft_consensus.cc:3037] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:14.816368 20800 raft_consensus.cc:504] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:14.818853 20800 leader_election.cc:288] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:14.734618 20703 consensus_queue.cc:1035] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.065s
I20260807 03:39:14.801916 20862 raft_consensus.cc:482] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:14.822041 20862 raft_consensus.cc:504] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:14.810582 18298 raft_consensus.cc:1226] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 4. Current term is 5. Ops: []
I20260807 03:39:14.812702 20173 leader_election.cc:288] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:14.816740 18305 raft_consensus.cc:1261] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:39:14.793135 18190 raft_consensus.cc:1226] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:14.830726 18190 raft_consensus.cc:1226] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 5. Current term is 6. Ops: []
W20260807 03:39:14.833459 20721 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260807 03:39:14.833529 18305 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57268 (request call id 1078) took 1411 ms. Trace:
I20260807 03:39:14.802723 20842 raft_consensus.cc:482] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:14.803179 20804 raft_consensus.cc:482] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:14.805182 20722 raft_consensus.cc:729] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:14.825002 20678 raft_consensus.cc:3037] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:14.838980 20691 raft_consensus.cc:2726] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 03:39:14.855381 20722 consensus_queue.cc:260] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:14.849131 20842 raft_consensus.cc:504] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:14.801337 20857 raft_consensus.cc:504] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:14.856768 20498 raft_consensus.cc:3032] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Stepping down as leader of term 5
I20260807 03:39:14.858660 20765 raft_consensus.cc:504] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:14.858974 20880 raft_consensus.cc:504] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:14.834051 18305 rpcz_store.cc:276] 0807 03:39:13.422394 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:13.424231 (+ 1837us) service_pool.cc:227] Handling call
0807 03:39:13.424589 (+ 358us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:14.326268 (+901679us) spinlock_profiling.cc:243] Waited 277 ms on lock 0x7b18000122a0. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921fa856 00007fe99214cf7e 00007fe9921b6367 00007fe9921b7b4e 00007fe9921c1c89 00007fe9921c2556 00007fe9921beedf 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684
0807 03:39:14.775503 (+449235us) spinlock_profiling.cc:243] Waited 431 ms on lock 0x7b58001004b0. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe99213af63 00007fe99219eaba 00007fe9921b63af 00007fe9921b7b4e 00007fe9921c1c89 00007fe9921c2556 00007fe9921beedf 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684
0807 03:39:14.833254 (+ 57751us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:14.833478 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1628714112}
I20260807 03:39:14.863248 18183 raft_consensus.cc:2454] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:14.863287 18191 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44660 (request call id 1284) took 1599 ms. Trace:
I20260807 03:39:14.848299 20800 raft_consensus.cc:482] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:14.890044 18141 leader_election.cc:302] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:14.893332 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.900954 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.898952 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.903096 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.901539 18295 raft_consensus.cc:2454] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 5.
I20260807 03:39:14.905237 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.906297 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.907300 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.907756 18311 raft_consensus.cc:1226] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 4. Current term is 5. Ops: []
I20260807 03:39:14.908223 20746 raft_consensus.cc:2781] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:14.908629 20746 raft_consensus.cc:482] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:14.908963 20746 raft_consensus.cc:3037] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:14.908989 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.910461 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.911643 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:14.913004 18183 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44660 (request call id 1254) took 3247 ms (client timeout 2142 ms). Trace:
W20260807 03:39:14.913398 18183 rpcz_store.cc:269] 0807 03:39:11.665703 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:11.665887 (+ 184us) service_pool.cc:227] Handling call
0807 03:39:14.912780 (+3246893us) spinlock_profiling.cc:243] Waited 3.18 s on lock 0x7b540016f3d4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:14.912882 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7309225344}
I20260807 03:39:14.915161 18141 leader_election.cc:302] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:14.915885 20813 raft_consensus.cc:2781] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:14.916326 20813 raft_consensus.cc:482] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:14.916723 20813 raft_consensus.cc:3037] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:14.916985 18141 leader_election.cc:302] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:14.913235 20800 raft_consensus.cc:504] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:14.917997 18141 leader_election.cc:302] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:14.918824 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.866382 20694 leader_election.cc:288] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:14.921301 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.921947 18295 raft_consensus.cc:2454] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:39:14.867437 20498 raft_consensus.cc:729] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:14.888989 20722 raft_consensus.cc:3037] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:14.922901 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.923405 18073 raft_consensus.cc:2379] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 6: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:14.900132 20882 raft_consensus.cc:482] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:14.924984 18295 raft_consensus.cc:2454] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 1.
I20260807 03:39:14.899179 18191 rpcz_store.cc:276] 0807 03:39:13.273666 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:13.278308 (+ 4642us) service_pool.cc:227] Handling call
0807 03:39:13.282521 (+ 4213us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:13.751101 (+468580us) spinlock_profiling.cc:243] Waited 2.15 ms on lock 0x7b180029bcc0. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe9981368b1 00007fe998143661 00007fe991e4e82c 00007fe98f2799dc 00007fe9980b0a39 00007fe98f2968bb 00007fe98f294cbf 00007fe98f294155 00007fe9921b73d0 00007fe9921b7f6b 00007fe9921c278b
0807 03:39:13.751299 (+ 198us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:13.751335 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:39:13.751862 (+ 527us) log.cc:842] Serialized 26 byte log entry
0807 03:39:14.475958 (+724096us) raft_consensus.cc:1631] Marking committed up to 4
0807 03:39:14.847381 (+371423us) log.cc:842] Serialized 14 byte log entry
0807 03:39:14.848782 (+ 1401us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:14.848894 (+ 112us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:14.848954 (+ 60us) raft_consensus.cc:1671] finished
0807 03:39:14.863103 (+ 14149us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:14.863235 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4407,"prepare.queue_time_us":339088,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":32,"raft.queue_time_us":32651,"raft.run_cpu_time_us":287,"raft.run_wall_time_us":13166,"spinlock_wait_cycles":4942080,"thread_start_us":1022220,"threads_started":3,"wal-append.queue_time_us":916969}
I20260807 03:39:14.925504 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.943035 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:14.944279 18295 raft_consensus.cc:2454] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:39:14.852124 20804 raft_consensus.cc:504] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:14.921130 20123 raft_consensus.cc:2781] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:39:14.961351 20939 leader_election.cc:288] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:14.964210 20857 leader_election.cc:288] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:14.475325 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:39:14.950652 20882 raft_consensus.cc:504] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:14.969218 18187 raft_consensus.cc:1261] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 03:39:14.971498 20882 leader_election.cc:288] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:14.924237 20124 raft_consensus.cc:2781] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:14.972391 20124 raft_consensus.cc:482] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:14.972772 20124 raft_consensus.cc:3037] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:14.973814 18187 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44660 (request call id 1291) took 1689 ms. Trace:
I20260807 03:39:14.974170 18187 rpcz_store.cc:276] 0807 03:39:13.287060 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:13.618707 (+331647us) service_pool.cc:227] Handling call
0807 03:39:14.915566 (+1296859us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:14.969542 (+ 53976us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:14.973008 (+ 3466us) spinlock_profiling.cc:243] Waited 53.3 ms on lock 0x7b540016f3d4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:39:14.973629 (+ 621us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b540016f3d0. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bea09 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:14.973764 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3104234112}
I20260807 03:39:14.975575 20862 leader_election.cc:288] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:14.977059 20880 leader_election.cc:288] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:14.964224 20765 leader_election.cc:288] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:14.979213 20804 leader_election.cc:288] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:14.983527 17920 catalog_manager.cc:5495] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:14.984169 20842 leader_election.cc:288] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:14.998301 18141 leader_election.cc:302] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
W20260807 03:39:15.000649 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 456881us
I20260807 03:39:15.000934 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:14.958074 20123 raft_consensus.cc:482] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:15.018114 17920 catalog_manager.cc:5495] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 1 to 3, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132). New cstate: current_term: 3 leader_uuid: "5c9b5062bdfa46af9396f1a85025a331" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:15.037727 18141 leader_election.cc:302] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:15.020183 17920 catalog_manager.cc:5495] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 3 to 4, leader changed from d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 4 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:15.038635 18141 leader_election.cc:302] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:15.039361 17914 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39510 (request call id 140) took 10277 ms. Trace:
I20260807 03:39:15.001294 20498 consensus_queue.cc:260] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:15.039870 17914 rpcz_store.cc:276] 0807 03:39:04.761549 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:04.761733 (+ 184us) service_pool.cc:227] Handling call
0807 03:39:15.039270 (+10277537us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 03:39:11.785399 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 03:39:11.881283 (+ 95884us) write_op.cc:432] Acquiring schema lock in shared mode
0807 03:39:11.881319 (+ 36us) write_op.cc:435] Acquired schema lock
0807 03:39:11.881333 (+ 14us) tablet.cc:660] Decoding operations
0807 03:39:11.881734 (+ 401us) write_op.cc:620] Acquiring the partition lock for write op
0807 03:39:11.881782 (+ 48us) write_op.cc:641] Partition lock acquired for write op
0807 03:39:11.881813 (+ 31us) tablet.cc:683] Acquiring locks for 2 operations
0807 03:39:11.882165 (+ 352us) tablet.cc:699] Row locks acquired
0807 03:39:11.882187 (+ 22us) write_op.cc:260] PREPARE: finished
0807 03:39:11.882322 (+ 135us) write_op.cc:270] Start()
0807 03:39:11.882436 (+ 114us) write_op.cc:276] Timestamp: P: 1786073951882279 usec, L: 0
0807 03:39:11.882453 (+ 17us) op_driver.cc:331] REPLICATION: starting
0807 03:39:11.883091 (+ 638us) log.cc:842] Serialized 929 byte log entry
0807 03:39:12.269892 (+386801us) op_driver.cc:444] REPLICATION: finished
0807 03:39:12.564655 (+294763us) write_op.cc:301] APPLY: starting
0807 03:39:12.569382 (+ 4727us) tablet.cc:1365] starting BulkCheckPresence
0807 03:39:12.575462 (+ 6080us) tablet.cc:1368] finished BulkCheckPresence
0807 03:39:12.593823 (+ 18361us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 03:39:12.627052 (+ 33229us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 03:39:12.633931 (+ 6879us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0807 03:39:12.638667 (+ 4736us) write_op.cc:312] APPLY: finished
0807 03:39:12.650663 (+ 11996us) log.cc:842] Serialized 28 byte log entry
0807 03:39:12.671561 (+ 20898us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:39:12.810674 (+139113us) write_op.cc:454] Released schema lock
0807 03:39:12.828182 (+ 17508us) write_op.cc:341] FINISH: Updating metrics
0807 03:39:12.864562 (+ 36380us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:39:12.864588 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":194550,"reported_tablets":23,"child_traces":[["op",{"apply.queue_time_us":259616,"apply.run_cpu_time_us":3166,"apply.run_wall_time_us":311716,"num_ops":2,"prepare.queue_time_us":160863,"prepare.run_cpu_time_us":10357,"prepare.run_wall_time_us":153744,"raft.queue_time_us":146972,"raft.run_cpu_time_us":51181,"raft.run_wall_time_us":402328,"replication_time_us":381179,"thread_start_us":452764,"threads_started":4,"wal-append.queue_time_us":65514,"wal-append.run_cpu_time_us":71034,"wal-append.run_wall_time_us":2216569}]]}
I20260807 03:39:15.044221 20498 raft_consensus.cc:3037] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:15.045044 20800 leader_election.cc:288] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:15.004407 18140 leader_election.cc:302] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:15.023056 20917 raft_consensus.cc:2781] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260807 03:39:15.063480 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 58967.7us
I20260807 03:39:15.008579 20123 raft_consensus.cc:3037] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:15.008998 20882 consensus_queue.cc:1046] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.054s
I20260807 03:39:15.048120 20651 raft_consensus.cc:2781] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:15.050899 20798 leader_election.cc:288] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:15.067615 20912 consensus_queue.cc:1035] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20260807 03:39:15.062714 20917 raft_consensus.cc:482] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:15.075521 17920 catalog_manager.cc:5495] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 3 to 4, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 4 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:39:15.153264 17920 catalog_manager.cc:5495] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 1 to 3, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 3 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
W20260807 03:39:15.142290 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 70541.7us
I20260807 03:39:15.068842 20651 raft_consensus.cc:482] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:15.037536 20658 consensus_queue.cc:1035] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20260807 03:39:15.202569 17920 catalog_manager.cc:5495] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 2 to 3, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 3 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:39:15.205013 17920 catalog_manager.cc:5495] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 3 to 5, leader changed from d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 5 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:15.209772 17920 catalog_manager.cc:5495] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
W20260807 03:39:15.220966 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 66080.8us
I20260807 03:39:15.222057 17920 catalog_manager.cc:5495] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 3 to 4, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 4 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:39:15.224792 17920 catalog_manager.cc:5495] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 1 to 3, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132). New cstate: current_term: 3 leader_uuid: "5c9b5062bdfa46af9396f1a85025a331" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
W20260807 03:39:15.133728 19239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 62148.7us
I20260807 03:39:15.226804 20822 consensus_queue.cc:1046] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:39:15.229102 20857 consensus_queue.cc:1035] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:39:15.151823 20583 raft_consensus.cc:504] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:15.229931 18248 leader_election.cc:302] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
W20260807 03:39:15.285952 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 56094.4us
I20260807 03:39:15.236557 20909 raft_consensus.cc:3032] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 03:39:15.145133 20653 raft_consensus.cc:2781] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260807 03:39:15.239881 18027 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.512s user 0.020s sys 0.024s
I20260807 03:39:15.201473 20917 raft_consensus.cc:3037] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
W20260807 03:39:15.241747 18189 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.524s user 0.022s sys 0.096s
I20260807 03:39:15.200973 20651 raft_consensus.cc:3037] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
W20260807 03:39:15.240881 20888 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.544s user 0.000s sys 0.026s
W20260807 03:39:15.256250 18249 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.533s user 0.019s sys 0.057s
W20260807 03:39:15.315366 18249 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.593s user 0.019s sys 0.057s
W20260807 03:39:15.315690 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 593025us
I20260807 03:39:15.256592 18300 raft_consensus.cc:1261] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20260807 03:39:15.260105 18312 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.012s sys 0.012s
I20260807 03:39:15.252820 20813 raft_consensus.cc:504] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:15.261076 20804 raft_consensus.cc:482] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:15.329609 20816 consensus_queue.cc:1046] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 03:39:15.330075 20804 raft_consensus.cc:504] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:15.330674 20816 raft_consensus.cc:3032] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 03:39:15.331089 20816 raft_consensus.cc:729] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:15.316761 20854 raft_consensus.cc:482] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:15.332379 20816 consensus_queue.cc:260] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:15.333069 20854 raft_consensus.cc:504] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:15.333811 20816 raft_consensus.cc:3037] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:15.338195 20583 leader_election.cc:288] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:15.261772 18185 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.533s user 0.039s sys 0.033s
W20260807 03:39:15.340276 18185 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.611s user 0.039s sys 0.033s
I20260807 03:39:15.272940 17920 catalog_manager.cc:5495] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 1 to 2, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 2 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
W20260807 03:39:15.287163 18027 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.559s user 0.021s sys 0.024s
W20260807 03:39:15.341991 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 622874us
W20260807 03:39:15.292431 18189 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.574s user 0.022s sys 0.096s
I20260807 03:39:15.292793 20909 raft_consensus.cc:729] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:15.311896 20543 raft_consensus.cc:2726] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:15.318022 18249 leader_election.cc:302] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:15.292640 20653 raft_consensus.cc:482] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:15.347952 18249 leader_election.cc:302] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
W20260807 03:39:15.348717 18310 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.609s user 0.009s sys 0.018s
W20260807 03:39:15.318531 18312 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.572s user 0.012s sys 0.012s
W20260807 03:39:15.349071 18310 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.609s user 0.010s sys 0.018s
I20260807 03:39:15.323825 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:15.327104 18080 raft_consensus.cc:1226] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:15.327244 18248 leader_election.cc:302] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:15.350803 18248 leader_election.cc:302] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:15.329373 20813 leader_election.cc:288] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:15.342656 17920 catalog_manager.cc:5495] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 1 to 2, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 2 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:39:15.354374 20721 raft_consensus.cc:2781] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 03:39:15.345786 18027 leader_election.cc:302] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:15.354779 20721 raft_consensus.cc:482] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:15.355198 20721 raft_consensus.cc:3037] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:15.355350 18070 raft_consensus.cc:1261] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 03:39:15.355489 18027 leader_election.cc:302] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:15.355067 17920 catalog_manager.cc:5495] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 3 to 5, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 5 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:15.356561 18027 leader_election.cc:302] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:15.320195 18073 raft_consensus.cc:1226] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 5. Current term is 6. Ops: []
W20260807 03:39:15.316813 20888 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.619s user 0.000s sys 0.026s
I20260807 03:39:15.349598 18082 raft_consensus.cc:2454] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:39:15.353729 20909 consensus_queue.cc:260] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:15.353394 18249 leader_election.cc:302] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:15.357674 18027 leader_election.cc:302] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:15.362022 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:15.360389 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:15.362792 18081 raft_consensus.cc:2454] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:39:15.317540 20798 raft_consensus.cc:482] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:15.358562 17920 catalog_manager.cc:5495] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 2 to 4, leader changed from d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 4 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:15.363341 18027 leader_election.cc:302] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:15.364137 20789 raft_consensus.cc:2781] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:39:15.364255 20703 consensus_queue.cc:1035] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:39:15.364590 20789 raft_consensus.cc:482] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:15.364634 18027 leader_election.cc:302] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:15.364912 20880 raft_consensus.cc:2726] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:15.365078 20789 raft_consensus.cc:3037] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:15.365576 20694 raft_consensus.cc:2781] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:15.365989 20880 raft_consensus.cc:2741] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260807 03:39:15.366046 20694 raft_consensus.cc:482] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:15.366588 20694 raft_consensus.cc:3037] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:15.367568 18027 leader_election.cc:302] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:15.368396 20880 raft_consensus.cc:2726] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:15.339984 20746 raft_consensus.cc:504] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:15.370296 18027 leader_election.cc:302] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:15.371065 20880 raft_consensus.cc:2781] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:39:15.371426 20880 raft_consensus.cc:482] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:15.356827 20653 raft_consensus.cc:3037] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:15.371835 20880 raft_consensus.cc:3037] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:15.362059 20862 raft_consensus.cc:2726] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 03:39:15.336020 20854 leader_election.cc:288] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:15.375106 20543 raft_consensus.cc:2781] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:39:15.389910 20543 raft_consensus.cc:482] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:15.389667 20917 raft_consensus.cc:504] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:15.390341 20543 raft_consensus.cc:3037] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:15.376976 17920 catalog_manager.cc:5495] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 3 to 5, leader changed from d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 5 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:15.370378 20746 leader_election.cc:288] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:15.372052 20123 raft_consensus.cc:504] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:15.359313 20699 raft_consensus.cc:2781] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:39:15.359513 20822 raft_consensus.cc:2781] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:15.392243 17920 catalog_manager.cc:5495] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 1 to 2, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 2 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:15.393436 20699 raft_consensus.cc:482] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:15.360369 20804 leader_election.cc:288] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:15.394209 17920 catalog_manager.cc:5495] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 1 to 3, leader changed from d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 3 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:15.395485 20699 raft_consensus.cc:3037] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:15.364773 20765 raft_consensus.cc:2726] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:15.379350 18298 raft_consensus.cc:1261] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260807 03:39:15.398973 18298 pending_rounds.cc:76] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49: Aborting all ops after (but not including) 2
I20260807 03:39:15.399204 18298 pending_rounds.cc:97] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260807 03:39:15.399462 18298 raft_consensus.cc:2866] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 03:39:15.379725 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:15.379931 18027 leader_election.cc:302] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:15.382129 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:15.402333 18075 raft_consensus.cc:2396] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 03:39:15.374204 18249 leader_election.cc:302] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:15.385926 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:15.406029 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:15.406816 18082 raft_consensus.cc:2454] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:39:15.411324 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:15.412117 18080 raft_consensus.cc:2454] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 6.
I20260807 03:39:15.413906 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:15.418336 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:15.420378 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:15.421149 18071 raft_consensus.cc:2454] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:39:15.376415 18248 leader_election.cc:302] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:15.408414 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:15.409343 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ca0223f47b46fd955704eba5259370" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:15.438934 18079 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.901s user 0.022s sys 0.036s
W20260807 03:39:15.493642 18079 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.956s user 0.024s sys 0.040s
I20260807 03:39:15.446751 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:15.447178 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:15.419130 18072 raft_consensus.cc:2454] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 5.
I20260807 03:39:15.373329 20789 raft_consensus.cc:504] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:15.435103 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 55187.5us
I20260807 03:39:15.394340 20822 raft_consensus.cc:482] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:15.505939 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:15.500753 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 124407us
I20260807 03:39:15.502718 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:15.506832 18072 raft_consensus.cc:2454] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:39:15.507938 20722 consensus_queue.cc:1035] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 03:39:15.511484 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:15.366074 20909 raft_consensus.cc:3037] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:15.503521 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:15.515415 18248 leader_election.cc:302] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:15.517577 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:15.383339 20805 consensus_meta.cc:287] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.633s user 0.007s sys 0.005s
I20260807 03:39:15.497054 18080 raft_consensus.cc:2454] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:39:15.528230 18311 raft_consensus.cc:2454] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 5.
I20260807 03:39:15.494469 20765 raft_consensus.cc:2781] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:39:15.495388 20277 raft_consensus.cc:2781] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 03:39:15.502012 20746 raft_consensus.cc:482] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:15.530169 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:15.531925 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:15.499958 20939 raft_consensus.cc:2781] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:15.387298 20511 raft_consensus.cc:3032] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Stepping down as leader of term 7
I20260807 03:39:15.523046 18070 raft_consensus.cc:2454] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
W20260807 03:39:15.520855 18180 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.763s user 0.009s sys 0.035s
W20260807 03:39:15.580646 18180 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.823s user 0.009s sys 0.035s
I20260807 03:39:15.382153 20124 raft_consensus.cc:504] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:15.584379 18305 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.622s user 0.016s sys 0.013s
W20260807 03:39:15.555910 18309 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.714s user 0.015s sys 0.042s
I20260807 03:39:15.512755 20703 raft_consensus.cc:3037] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:15.563313 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:15.378415 20880 raft_consensus.cc:504] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:15.564406 20853 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.721s user 0.016s sys 0.036s
I20260807 03:39:15.567112 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:15.570277 20917 leader_election.cc:288] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:15.526638 19617 consensus_queue.cc:1035] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.049s
I20260807 03:39:15.573235 20746 raft_consensus.cc:504] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:15.574595 20123 leader_election.cc:288] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:15.575562 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:15.582249 19238 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.665s user 0.016s sys 0.017s
I20260807 03:39:15.379633 20651 raft_consensus.cc:504] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:15.516361 20822 raft_consensus.cc:3037] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:15.527642 20498 raft_consensus.cc:482] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:15.527519 20722 raft_consensus.cc:2781] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:39:15.546836 18248 leader_election.cc:302] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:15.555533 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:15.584761 18305 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.622s user 0.016s sys 0.013s
W20260807 03:39:15.584903 18309 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.743s user 0.016s sys 0.042s
I20260807 03:39:15.585040 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:15.534111 20765 raft_consensus.cc:482] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:15.587018 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:15.529337 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:15.597160 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
W20260807 03:39:15.610033 18295 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.642s user 0.018s sys 0.039s
W20260807 03:39:15.627086 18295 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.659s user 0.018s sys 0.039s
I20260807 03:39:15.609340 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:15.577025 20277 raft_consensus.cc:686] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:15.383607 20798 raft_consensus.cc:504] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:15.579790 20511 raft_consensus.cc:729] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:15.613163 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:15.635417 18298 raft_consensus.cc:2454] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 6.
I20260807 03:39:15.613077 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:15.617620 18193 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.569s user 0.015s sys 0.026s
W20260807 03:39:15.606150 20853 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.760s user 0.016s sys 0.036s
I20260807 03:39:15.639021 20277 consensus_queue.cc:237] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:15.602702 18027 leader_election.cc:302] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:15.640584 20511 consensus_queue.cc:260] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:15.641461 18027 leader_election.cc:302] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:15.641867 20511 raft_consensus.cc:3037] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:15.642642 18027 leader_election.cc:302] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
W20260807 03:39:15.643646 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 96922.3us
I20260807 03:39:15.627579 18072 raft_consensus.cc:2454] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
W20260807 03:39:15.614833 19238 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.712s user 0.017s sys 0.017s
I20260807 03:39:15.647667 20853 raft_consensus.cc:2781] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:15.648038 20853 raft_consensus.cc:482] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:15.648447 20853 raft_consensus.cc:3037] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:15.648869 20961 raft_consensus.cc:2781] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260807 03:39:15.637737 18193 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.589s user 0.015s sys 0.026s
I20260807 03:39:15.638795 18073 raft_consensus.cc:1261] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:39:15.625912 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:15.643754 18027 leader_election.cc:302] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:15.649369 20961 raft_consensus.cc:482] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:15.652442 20961 raft_consensus.cc:3037] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:15.621826 20722 raft_consensus.cc:482] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:15.647395 20583 raft_consensus.cc:2726] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:15.657789 18081 raft_consensus.cc:2454] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:39:15.644742 18248 leader_election.cc:302] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:15.644932 18310 raft_consensus.cc:2454] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 4.
I20260807 03:39:15.653983 18193 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44660 (request call id 1315) took 1023 ms. Trace:
I20260807 03:39:15.652457 19292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
W20260807 03:39:15.618834 20684 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.626s user 0.000s sys 0.031s
W20260807 03:39:15.664358 20684 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.702s user 0.000s sys 0.031s
I20260807 03:39:15.616730 20699 raft_consensus.cc:504] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:15.629887 20765 raft_consensus.cc:3037] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:15.664520 20939 raft_consensus.cc:482] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:15.664971 20939 raft_consensus.cc:3037] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:15.652833 20842 raft_consensus.cc:482] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
W20260807 03:39:15.654287 18184 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.509s user 0.035s sys 0.044s
I20260807 03:39:15.666119 20699 leader_election.cc:288] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
W20260807 03:39:15.667508 18184 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.523s user 0.035s sys 0.044s
I20260807 03:39:15.626689 20962 raft_consensus.cc:482] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:15.667865 20651 leader_election.cc:288] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:15.669162 18188 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44660 (request call id 1314) took 1038 ms. Trace:
I20260807 03:39:15.674453 18188 rpcz_store.cc:276] 0807 03:39:14.630637 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:14.634162 (+ 3525us) service_pool.cc:227] Handling call
0807 03:39:15.668973 (+1034811us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b54003c8f54. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:15.669108 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2375152384}
I20260807 03:39:15.669480 18073 raft_consensus.cc:1226] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 1. Current term is 2. Ops: []
I20260807 03:39:15.652995 20691 raft_consensus.cc:2726] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 03:39:15.670936 18248 leader_election.cc:302] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:15.677172 18248 leader_election.cc:302] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:15.670930 18184 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44660 (request call id 1321) took 1037 ms. Trace:
I20260807 03:39:15.670212 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:15.626499 20498 raft_consensus.cc:504] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:15.676067 20962 raft_consensus.cc:504] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:15.676275 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 454606us
I20260807 03:39:15.695295 18248 leader_election.cc:302] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:15.656471 20800 raft_consensus.cc:2726] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:15.691740 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:15.696521 18248 leader_election.cc:302] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:15.690908 20854 raft_consensus.cc:482] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:15.690779 20857 raft_consensus.cc:2726] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:15.691958 20805 raft_consensus.cc:2781] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:39:15.703086 18188 raft_consensus.cc:1261] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:39:15.703238 20805 raft_consensus.cc:482] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:15.703675 20805 raft_consensus.cc:3037] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:15.704671 20939 raft_consensus.cc:504] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:15.705322 20976 log.cc:924] Time spent T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.071s user 0.002s sys 0.000s
I20260807 03:39:15.692601 18184 rpcz_store.cc:276] 0807 03:39:14.632813 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:14.682013 (+ 49200us) service_pool.cc:227] Handling call
0807 03:39:14.682419 (+ 406us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:14.682643 (+ 224us) raft_consensus.cc:1520] Early marking committed up to index 2
0807 03:39:15.144626 (+461983us) log.cc:842] Serialized 14 byte log entry
0807 03:39:15.670232 (+525606us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:15.670282 (+ 50us) raft_consensus.cc:1631] Marking committed up to 2
0807 03:39:15.670330 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:15.670389 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:15.670539 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":132862,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"raft.queue_time_us":71,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":4365,"thread_start_us":970015,"threads_started":2}
I20260807 03:39:15.700474 20678 raft_consensus.cc:2781] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 03:39:15.709494 20678 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:15.709893 20678 raft_consensus.cc:3037] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:15.701709 18248 leader_election.cc:302] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:15.659683 20722 raft_consensus.cc:3037] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:15.701087 20900 raft_consensus.cc:2726] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:15.671522 20699 raft_consensus.cc:2726] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:15.717772 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:15.718386 18081 raft_consensus.cc:2454] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:39:15.670665 20975 raft_consensus.cc:2781] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader election won for term 6
I20260807 03:39:15.724329 18188 raft_consensus.cc:1203] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Deduplicated request from leader. Original: 1.1->[2.2-3.3] Dedup: 2.2->[3.3-3.3]
I20260807 03:39:15.724843 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:15.661111 18193 rpcz_store.cc:276] 0807 03:39:14.630975 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:14.635340 (+ 4365us) service_pool.cc:227] Handling call
0807 03:39:14.635680 (+ 340us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:14.635920 (+ 240us) raft_consensus.cc:1520] Early marking committed up to index 3
0807 03:39:15.048724 (+412804us) log.cc:842] Serialized 14 byte log entry
0807 03:39:15.653569 (+604845us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:15.653617 (+ 48us) raft_consensus.cc:1631] Marking committed up to 3
0807 03:39:15.653658 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:15.653824 (+ 166us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:15.653930 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":179368,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":1143,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":3774,"thread_start_us":989910,"threads_started":2}
I20260807 03:39:15.666622 20842 raft_consensus.cc:504] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:15.724979 18188 raft_consensus.cc:1226] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 3. Current term is 4. Ops: [2.2-3.3]
W20260807 03:39:15.711266 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 646660us
I20260807 03:39:15.730676 20939 leader_election.cc:288] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:15.720963 20816 raft_consensus.cc:2781] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:15.731442 20816 raft_consensus.cc:482] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:15.731784 20816 raft_consensus.cc:3037] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:15.732463 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:15.733057 18081 raft_consensus.cc:2454] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 5.
I20260807 03:39:15.701763 20789 leader_election.cc:288] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:15.721163 20620 consensus_queue.cc:1035] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
W20260807 03:39:15.741338 18297 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.735s user 0.033s sys 0.058s
W20260807 03:39:15.762739 18297 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.760s user 0.033s sys 0.059s
I20260807 03:39:15.754134 18248 leader_election.cc:302] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:15.757076 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:39:15.766592 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 50343.9us
I20260807 03:39:15.757133 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:15.757807 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:15.723423 20798 leader_election.cc:288] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:15.773257 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:15.775101 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:15.808636 18190 raft_consensus.cc:2454] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 5.
I20260807 03:39:15.768461 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:15.779325 20960 raft_consensus.cc:2726] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:15.702260 20653 raft_consensus.cc:504] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:15.710412 20854 raft_consensus.cc:504] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:15.721474 20651 consensus_queue.cc:1035] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
I20260807 03:39:15.757899 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:15.774829 20880 leader_election.cc:288] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:15.796564 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:15.791716 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:15.792448 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:15.772962 20746 leader_election.cc:288] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:15.791348 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:15.805397 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:15.805923 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:15.758442 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:15.738626 20842 leader_election.cc:288] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:15.811288 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:15.735442 20975 raft_consensus.cc:686] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
W20260807 03:39:15.838114 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 186378us
I20260807 03:39:15.830308 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:15.772611 20888 consensus_queue.cc:1035] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
I20260807 03:39:15.770736 20900 consensus_queue.cc:1035] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
I20260807 03:39:15.837337 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:15.816426 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:39:15.826617 18003 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica 66a36e1ac8e945e893668e282d394c49 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:15.845006 18309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57268 (request call id 1106) took 1049 ms. Trace:
I20260807 03:39:15.828037 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:15.790943 20658 raft_consensus.cc:482] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:15.834087 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:15.791057 20123 raft_consensus.cc:482] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:15.782040 20699 consensus_queue.cc:1035] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20260807 03:39:15.829437 18075 raft_consensus.cc:2379] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 6: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:15.850032 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:15.813767 18185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44660 (request call id 1323) took 1180 ms. Trace:
W20260807 03:39:15.830291 20969 log.cc:924] Time spent T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.094s user 0.000s sys 0.001s
I20260807 03:39:15.840592 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:15.843160 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:15.857036 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:15.858053 18183 raft_consensus.cc:2379] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 5: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:15.846304 18309 rpcz_store.cc:276] 0807 03:39:14.795572 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:14.795780 (+ 208us) service_pool.cc:227] Handling call
0807 03:39:14.807090 (+ 11310us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:14.807339 (+ 249us) raft_consensus.cc:1520] Early marking committed up to index 3
0807 03:39:15.622735 (+815396us) log.cc:842] Serialized 14 byte log entry
0807 03:39:15.844593 (+221858us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:15.844658 (+ 65us) raft_consensus.cc:1631] Marking committed up to 3
0807 03:39:15.844700 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:15.844766 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:15.844960 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3798,"prepare.queue_time_us":39966,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":1879,"thread_start_us":993643,"threads_started":2}
I20260807 03:39:15.858992 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:15.845811 18188 raft_consensus.cc:3037] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
W20260807 03:39:15.829788 20959 log.cc:924] Time spent T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.174s user 0.000s sys 0.001s
W20260807 03:39:15.828858 20987 log.cc:924] Time spent T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
I20260807 03:39:15.851631 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:15.835578 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:15.848932 20965 log.cc:924] Time spent T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.179s user 0.000s sys 0.000s
I20260807 03:39:15.857223 18310 raft_consensus.cc:3037] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
W20260807 03:39:15.855144 20978 log.cc:924] Time spent T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.157s user 0.000s sys 0.001s
I20260807 03:39:15.869431 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:15.847954 18027 leader_election.cc:302] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:15.872483 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:15.876271 18072 raft_consensus.cc:2454] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 5.
I20260807 03:39:15.851560 18185 rpcz_store.cc:276] 0807 03:39:14.633380 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:14.701817 (+ 68437us) service_pool.cc:227] Handling call
0807 03:39:14.702218 (+ 401us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:14.702561 (+ 343us) raft_consensus.cc:1520] Early marking committed up to index 3
0807 03:39:15.345450 (+642889us) log.cc:842] Serialized 14 byte log entry
0807 03:39:15.812632 (+467182us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:15.812682 (+ 50us) raft_consensus.cc:1631] Marking committed up to 3
0807 03:39:15.812722 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:15.813616 (+ 894us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:15.813720 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":560,"prepare.queue_time_us":113166,"prepare.run_cpu_time_us":26,"prepare.run_wall_time_us":27,"raft.queue_time_us":2484,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":17733,"thread_start_us":1077983,"threads_started":2}
I20260807 03:39:15.839718 20746 raft_consensus.cc:482] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:15.879058 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:15.881778 18188 raft_consensus.cc:2454] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 6.
I20260807 03:39:15.896724 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:15.898313 18184 raft_consensus.cc:2354] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 4. Current term is 5.
I20260807 03:39:15.898587 18075 raft_consensus.cc:2454] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 5.
I20260807 03:39:15.898984 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:15.887635 18310 raft_consensus.cc:2454] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 6.
I20260807 03:39:15.902643 18185 raft_consensus.cc:1226] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:15.903439 18187 raft_consensus.cc:1226] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:15.860452 20498 leader_election.cc:288] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:15.885540 20950 log.cc:924] Time spent T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.188s user 0.000s sys 0.001s
I20260807 03:39:15.903774 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:15.907720 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:39:15.907733 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 59916us
I20260807 03:39:15.908555 18183 raft_consensus.cc:2454] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 2.
I20260807 03:39:15.911721 18188 raft_consensus.cc:1226] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 4. Current term is 5. Ops: []
I20260807 03:39:15.890097 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:15.915958 18027 leader_election.cc:302] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:15.921319 20765 raft_consensus.cc:504] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:15.922493 20653 leader_election.cc:288] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:15.911826 20880 raft_consensus.cc:2781] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:39:15.930734 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:15.854346 20658 raft_consensus.cc:504] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:15.931425 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:15.853623 20123 raft_consensus.cc:504] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:15.937533 18309 raft_consensus.cc:3037] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:15.947418 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:15.950038 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:15.951437 18027 leader_election.cc:302] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:15.952714 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:15.906909 20746 raft_consensus.cc:504] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:15.961620 18190 raft_consensus.cc:2454] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 5.
I20260807 03:39:15.961762 18295 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57276 (request call id 833) took 1065 ms. Trace:
I20260807 03:39:15.963546 18295 rpcz_store.cc:276] 0807 03:39:14.896793 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:14.947733 (+ 50940us) service_pool.cc:227] Handling call
0807 03:39:14.948031 (+ 298us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:14.949525 (+ 1494us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:14.949558 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:39:14.957316 (+ 7758us) log.cc:842] Serialized 26 byte log entry
0807 03:39:15.627468 (+670152us) raft_consensus.cc:1631] Marking committed up to 2
0807 03:39:15.627503 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:15.627562 (+ 59us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:15.961465 (+333903us) raft_consensus.cc:1671] finished
0807 03:39:15.961496 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:15.961689 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2800,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":2730,"thread_start_us":659684,"threads_started":1,"wal-append.queue_time_us":742753}
I20260807 03:39:15.960875 18180 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44660 (request call id 1316) took 1329 ms. Trace:
I20260807 03:39:15.970582 18027 leader_election.cc:302] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:15.937345 20880 raft_consensus.cc:482] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:15.967366 18180 rpcz_store.cc:276] 0807 03:39:14.631280 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:14.636146 (+ 4866us) service_pool.cc:227] Handling call
0807 03:39:14.636519 (+ 373us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:14.656300 (+ 19781us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:14.656335 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:39:14.656779 (+ 444us) log.cc:842] Serialized 26 byte log entry
0807 03:39:14.757664 (+100885us) spinlock_profiling.cc:243] Waited 1 ms on lock 0x7b1800241180. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe9981368b1 00007fe998143661 00007fe98f279345 00007fe9980b0a39 00007fe98f2968bb 00007fe98f294cbf 00007fe98f296e22 00007fe991e405bb 00007fe991e46063 00007fe991e46411 00007fe9921935cf
0807 03:39:15.581100 (+823436us) raft_consensus.cc:1631] Marking committed up to 1
0807 03:39:15.581155 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:15.581243 (+ 88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:15.950259 (+369016us) raft_consensus.cc:1671] finished
0807 03:39:15.950291 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:15.960776 (+ 10485us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14537,"raft.queue_time_us":14592,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":93,"spinlock_wait_cycles":2300800,"thread_start_us":823207,"threads_started":1,"wal-append.queue_time_us":1015484}
I20260807 03:39:15.970801 19240 leader_election.cc:302] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:15.970894 20798 raft_consensus.cc:2781] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:15.990223 20798 raft_consensus.cc:482] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:15.990386 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:15.990662 20798 raft_consensus.cc:3037] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:15.991026 18070 raft_consensus.cc:3037] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
W20260807 03:39:15.995028 20983 log.cc:924] Time spent T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
I20260807 03:39:15.952401 20498 raft_consensus.cc:2781] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:39:15.974320 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:15.977380 20691 raft_consensus.cc:2726] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260807 03:39:15.996130 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 205253us
I20260807 03:39:15.973148 18079 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33254 (request call id 1220) took 1490 ms. Trace:
I20260807 03:39:15.938699 20816 raft_consensus.cc:504] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:15.985183 20990 log.cc:924] Time spent T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.174s user 0.000s sys 0.001s
I20260807 03:39:16.028929 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:16.029791 18071 raft_consensus.cc:3037] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:16.009295 18072 raft_consensus.cc:1261] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:39:16.009119 18070 raft_consensus.cc:2454] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 5.
I20260807 03:39:16.007182 18079 rpcz_store.cc:276] 0807 03:39:14.483167 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:14.483285 (+ 118us) service_pool.cc:227] Handling call
0807 03:39:14.483624 (+ 339us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:14.483989 (+ 365us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:14.484026 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:39:14.495204 (+ 11178us) log.cc:842] Serialized 26 byte log entry
0807 03:39:15.494087 (+998883us) raft_consensus.cc:1631] Marking committed up to 1
0807 03:39:15.494136 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:15.494207 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:15.972950 (+478743us) raft_consensus.cc:1671] finished
0807 03:39:15.972964 (+ 14us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:15.973068 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":10801,"raft.run_cpu_time_us":214,"raft.run_wall_time_us":22232,"thread_start_us":955971,"threads_started":1,"wal-append.queue_time_us":1132133}
I20260807 03:39:16.012717 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:16.005463 18309 raft_consensus.cc:2454] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 5.
I20260807 03:39:16.015183 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:15.972368 18027 leader_election.cc:302] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:15.981403 20880 raft_consensus.cc:3037] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:15.972738 20962 leader_election.cc:288] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:16.047991 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 51330.4us
W20260807 03:39:15.998685 20543 consensus_meta.cc:287] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.607s user 0.002s sys 0.009s
W20260807 03:39:16.052542 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 80224.9us
W20260807 03:39:16.012171 20992 log.cc:924] Time spent T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.171s user 0.000s sys 0.001s
I20260807 03:39:15.999277 20842 raft_consensus.cc:2726] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:15.995908 20854 leader_election.cc:288] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:15.998858 20498 raft_consensus.cc:482] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:16.057812 18293 raft_consensus.cc:2454] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 5.
I20260807 03:39:16.058882 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:16.051331 20721 consensus_meta.cc:287] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.696s user 0.008s sys 0.004s
I20260807 03:39:15.926851 20975 consensus_queue.cc:237] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.064639 18027 leader_election.cc:302] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:16.066111 18071 raft_consensus.cc:2454] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 5.
I20260807 03:39:16.062678 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:16.067094 18072 raft_consensus.cc:2454] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:39:16.067817 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:16.074775 18027 leader_election.cc:302] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:16.075469 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
W20260807 03:39:16.050307 19240 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 100148us
W20260807 03:39:16.062938 20694 consensus_meta.cc:287] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 0.696s user 0.008s sys 0.005s
I20260807 03:39:16.077286 18141 leader_election.cc:302] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:16.060149 20691 raft_consensus.cc:2781] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:39:16.081748 18141 leader_election.cc:302] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:16.077627 18071 raft_consensus.cc:3037] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:16.078099 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:16.082736 18189 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44660 (request call id 1319) took 1450 ms. Trace:
I20260807 03:39:16.086006 20765 leader_election.cc:288] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:16.086642 20765 raft_consensus.cc:2726] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 03:39:16.088802 18071 raft_consensus.cc:2454] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:39:16.092011 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:16.092757 18298 raft_consensus.cc:2454] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 4.
I20260807 03:39:16.093724 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:16.094981 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:16.088406 20699 raft_consensus.cc:482] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:16.096592 20699 raft_consensus.cc:504] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.083312 18189 rpcz_store.cc:276] 0807 03:39:14.632241 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:14.670810 (+ 38569us) service_pool.cc:227] Handling call
0807 03:39:14.677602 (+ 6792us) raft_consensus.cc:1457] Updating replica for 3 ops
0807 03:39:14.677961 (+ 359us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:14.677989 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0807 03:39:14.684936 (+ 6947us) log.cc:842] Serialized 78 byte log entry
0807 03:39:15.343129 (+658193us) raft_consensus.cc:1631] Marking committed up to 2
0807 03:39:15.353250 (+ 10121us) log.cc:842] Serialized 14 byte log entry
0807 03:39:15.353430 (+ 180us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:15.353488 (+ 58us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:16.082479 (+728991us) raft_consensus.cc:1671] finished
0807 03:39:16.082498 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:16.082664 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":15999,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":43,"raft.queue_time_us":773,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":7794,"thread_start_us":624886,"threads_started":1,"wal-append.queue_time_us":917184}
I20260807 03:39:16.068679 20498 raft_consensus.cc:3037] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:16.098796 18073 raft_consensus.cc:2379] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 4: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:16.098284 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:16.099628 18309 raft_consensus.cc:3037] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:16.099869 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:16.097419 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:16.101763 18072 raft_consensus.cc:2454] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:39:16.102123 18298 raft_consensus.cc:3037] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:16.076651 20862 raft_consensus.cc:2741] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election decision vote started in defunct term 4: won
I20260807 03:39:16.058976 20543 raft_consensus.cc:504] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.107296 18027 leader_election.cc:302] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:16.112697 18309 raft_consensus.cc:2454] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 5.
I20260807 03:39:16.114940 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
W20260807 03:39:16.122409 21003 log.cc:924] Time spent T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.111s user 0.000s sys 0.001s
I20260807 03:39:16.097748 20694 raft_consensus.cc:504] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:16.124723 21008 log.cc:924] Time spent T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.173s user 0.000s sys 0.000s
I20260807 03:39:16.124987 20694 leader_election.cc:288] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:16.098255 20912 raft_consensus.cc:2781] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader election won for term 6
W20260807 03:39:16.125916 21010 log.cc:924] Time spent T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20260807 03:39:16.126333 20912 raft_consensus.cc:686] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
W20260807 03:39:16.126840 21009 log.cc:924] Time spent T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.190s user 0.000s sys 0.001s
I20260807 03:39:16.127388 20912 consensus_queue.cc:237] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.097931 21015 raft_consensus.cc:2781] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 03:39:16.130242 21015 raft_consensus.cc:686] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:16.131078 21015 consensus_queue.cc:237] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.085407 20691 raft_consensus.cc:482] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:16.135834 18080 raft_consensus.cc:2379] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 4: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:16.135720 18027 leader_election.cc:302] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:16.136891 18027 leader_election.cc:302] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:16.137784 18027 leader_election.cc:302] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:16.138552 18027 leader_election.cc:302] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:16.139283 18027 leader_election.cc:302] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:16.114208 18141 leader_election.cc:302] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:16.070020 20721 raft_consensus.cc:504] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.153244 18305 raft_consensus.cc:2379] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 6: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
W20260807 03:39:16.151851 20703 consensus_meta.cc:287] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.548s user 0.004s sys 0.006s
I20260807 03:39:16.155334 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:16.158687 18309 raft_consensus.cc:2379] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 5: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
W20260807 03:39:16.145887 20988 log.cc:924] Time spent T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.237s user 0.000s sys 0.001s
I20260807 03:39:16.170966 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:16.172842 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:16.174991 18070 raft_consensus.cc:1226] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 1. Current term is 2. Ops: []
I20260807 03:39:16.175840 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:16.144742 20862 raft_consensus.cc:2726] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 03:39:16.141636 20691 raft_consensus.cc:3037] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:16.148255 20994 raft_consensus.cc:2781] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 03:39:16.148078 20694 raft_consensus.cc:2781] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:39:16.195195 18312 raft_consensus.cc:1226] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 7. Current term is 8. Ops: []
W20260807 03:39:16.195539 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 138330us
I20260807 03:39:16.196734 18248 leader_election.cc:302] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:16.157586 20765 raft_consensus.cc:2726] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:16.156445 20962 raft_consensus.cc:2726] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 03:39:16.206725 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:16.160841 20703 raft_consensus.cc:2726] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:16.161252 20804 raft_consensus.cc:2726] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 03:39:16.241138 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 101880us
I20260807 03:39:16.195147 20746 leader_election.cc:288] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:16.243882 18027 leader_election.cc:302] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:16.248265 18027 leader_election.cc:302] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:16.248521 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:16.244764 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:16.209689 20694 raft_consensus.cc:482] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:16.249548 18070 raft_consensus.cc:2454] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 6.
I20260807 03:39:16.245746 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:16.253620 18248 leader_election.cc:302] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
W20260807 03:39:16.242286 20678 consensus_meta.cc:287] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.532s user 0.006s sys 0.005s
I20260807 03:39:16.228236 20994 raft_consensus.cc:686] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:16.258008 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:16.243865 20511 raft_consensus.cc:2726] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 03:39:16.209111 20805 raft_consensus.cc:504] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.262490 18027 leader_election.cc:302] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:16.263031 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:16.255159 20694 raft_consensus.cc:3037] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:16.251871 20962 raft_consensus.cc:2726] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:16.255154 20816 leader_election.cc:288] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:16.273021 18027 leader_election.cc:302] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:16.291182 18027 leader_election.cc:302] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:16.292694 18027 leader_election.cc:398] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 election: Vote denied by peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057) with higher term. Message: Invalid argument: T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 4. Current term is 5.
W20260807 03:39:16.286012 20822 consensus_meta.cc:287] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.656s user 0.006s sys 0.004s
I20260807 03:39:16.286809 19292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:16.288187 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:16.302042 18071 raft_consensus.cc:2454] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:16.289655 18310 raft_consensus.cc:1226] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3. Ops: []
I20260807 03:39:16.295851 18298 raft_consensus.cc:2454] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:39:16.312443 18027 leader_election.cc:302] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:16.273240 20939 raft_consensus.cc:2726] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 03:39:16.318351 18071 raft_consensus.cc:1226] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
W20260807 03:39:16.297283 20967 log.cc:924] Time spent T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.644s user 0.005s sys 0.000s
I20260807 03:39:16.259831 20678 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.287050 21044 raft_consensus.cc:2726] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 03:39:16.301011 20765 raft_consensus.cc:2726] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 03:39:16.191439 21027 raft_consensus.cc:2726] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:16.300606 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:39:16.282447 20651 raft_consensus.cc:482] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:16.288679 20658 leader_election.cc:288] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:16.296365 20962 raft_consensus.cc:2726] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:16.317994 21035 raft_consensus.cc:2781] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:39:16.267333 18079 raft_consensus.cc:2379] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 6: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:16.316200 20543 leader_election.cc:288] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:16.377552 18298 raft_consensus.cc:1226] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 7. Current term is 8. Ops: []
W20260807 03:39:16.373690 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 61335us
W20260807 03:39:16.378558 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 133621us
I20260807 03:39:16.385353 18070 raft_consensus.cc:2454] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:16.386420 18141 leader_election.cc:302] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:16.377583 21035 raft_consensus.cc:482] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:16.312501 20822 raft_consensus.cc:504] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.396310 21035 raft_consensus.cc:3037] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:16.393714 20658 raft_consensus.cc:2781] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader election won for term 4
W20260807 03:39:16.397063 21045 log.cc:924] Time spent T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
I20260807 03:39:16.372515 21043 consensus_queue.cc:1046] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.071s
I20260807 03:39:16.375200 21044 raft_consensus.cc:2726] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260807 03:39:16.380589 20853 consensus_meta.cc:287] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 0.732s user 0.010s sys 0.003s
I20260807 03:39:16.403498 18249 leader_election.cc:302] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:16.429256 18249 leader_election.cc:302] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:16.405884 21061 consensus_queue.cc:1046] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 03:39:16.372239 20721 leader_election.cc:288] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:16.416514 20658 raft_consensus.cc:686] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
W20260807 03:39:16.417405 21000 log.cc:924] Time spent T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.501s user 0.000s sys 0.009s
I20260807 03:39:16.417846 20678 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:16.420272 20699 leader_election.cc:288] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:16.422358 18027 leader_election.cc:302] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
W20260807 03:39:16.392421 20967 log.cc:924] Time spent T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20260807 03:39:16.372337 20939 consensus_queue.cc:1046] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.039s
I20260807 03:39:16.421576 20123 leader_election.cc:288] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:16.428400 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 174884us
I20260807 03:39:16.432255 18300 raft_consensus.cc:1226] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:16.437242 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:16.449640 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:16.427831 20853 raft_consensus.cc:504] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.439692 21059 raft_consensus.cc:2781] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:16.462927 18073 raft_consensus.cc:1226] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260807 03:39:16.463784 18073 raft_consensus.cc:1226] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:16.450635 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:16.438407 20703 raft_consensus.cc:2781] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 03:39:16.453295 18141 leader_election.cc:302] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:16.377980 20994 consensus_queue.cc:237] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:16.457540 20722 consensus_meta.cc:287] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.729s user 0.010s sys 0.001s
I20260807 03:39:16.457333 20658 consensus_queue.cc:237] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.476251 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:16.444342 21049 raft_consensus.cc:2726] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 03:39:16.459149 20805 leader_election.cc:288] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:16.460407 20822 leader_election.cc:288] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:16.387204 20651 raft_consensus.cc:504] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.465173 21013 raft_consensus.cc:482] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:16.465773 18075 raft_consensus.cc:3037] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:16.469530 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:16.470520 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
W20260807 03:39:16.458076 20909 consensus_meta.cc:287] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 0.929s user 0.012s sys 0.000s
I20260807 03:39:16.465045 18312 raft_consensus.cc:2454] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 8.
I20260807 03:39:16.434401 21015 consensus_queue.cc:1046] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.007s
I20260807 03:39:16.478029 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:16.478446 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:16.479594 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:16.480131 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:16.482885 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:16.488019 18293 raft_consensus.cc:1226] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 6. Current term is 7. Ops: []
I20260807 03:39:16.488466 18189 raft_consensus.cc:2379] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 6: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:16.484269 18193 raft_consensus.cc:2379] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 5: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:16.486141 20853 leader_election.cc:288] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:16.462793 21059 raft_consensus.cc:482] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 03:39:16.483336 20961 consensus_meta.cc:287] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.828s user 0.005s sys 0.006s
I20260807 03:39:16.478400 20912 raft_consensus.cc:2726] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:16.490078 21059 raft_consensus.cc:3037] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:16.461131 20498 raft_consensus.cc:504] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.496165 18075 raft_consensus.cc:2454] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 6.
I20260807 03:39:16.497851 21035 raft_consensus.cc:504] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.512892 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:16.517035 18082 raft_consensus.cc:2379] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 3: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:16.519913 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:16.478529 20722 raft_consensus.cc:504] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.508203 20703 raft_consensus.cc:686] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 6 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:16.490119 20961 raft_consensus.cc:504] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:16.524339 18246 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.132:41381, user_credentials={real_user=slave}} blocked reactor thread for 46259.9us
I20260807 03:39:16.528440 18311 raft_consensus.cc:1226] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3. Ops: []
I20260807 03:39:16.523733 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:16.526998 18295 raft_consensus.cc:1226] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:16.530428 18141 leader_election.cc:302] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
W20260807 03:39:16.534219 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 83659us
I20260807 03:39:16.529006 18073 raft_consensus.cc:2454] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
I20260807 03:39:16.534618 18141 leader_election.cc:302] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:16.534919 18248 leader_election.cc:302] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:16.535451 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:16.536120 18073 raft_consensus.cc:3037] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:16.536022 18248 leader_election.cc:302] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:16.533635 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:16.525231 20703 consensus_queue.cc:237] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.538064 20498 leader_election.cc:288] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:16.525151 20722 leader_election.cc:288] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:16.539513 21011 raft_consensus.cc:2726] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260807 03:39:16.528082 20691 raft_consensus.cc:504] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.540400 20124 leader_election.cc:288] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:16.526053 20798 consensus_meta.cc:287] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 0.535s user 0.009s sys 0.000s
I20260807 03:39:16.547508 18180 raft_consensus.cc:1261] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 03:39:16.548966 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:16.549248 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:16.527541 20961 leader_election.cc:288] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:16.491994 21013 raft_consensus.cc:504] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.544626 21015 raft_consensus.cc:2781] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:39:16.534134 20822 raft_consensus.cc:482] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:16.560546 18248 leader_election.cc:302] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:16.560231 18309 raft_consensus.cc:1261] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260807 03:39:16.545701 20658 raft_consensus.cc:2781] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 03:39:16.574353 21017 consensus_queue.cc:1046] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260807 03:39:16.574858 20651 leader_election.cc:288] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:16.558308 20960 raft_consensus.cc:2781] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 03:39:16.557811 21056 raft_consensus.cc:482] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:16.559450 21015 raft_consensus.cc:482] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:16.576757 21015 raft_consensus.cc:3037] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:16.576642 21056 raft_consensus.cc:504] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.578770 21056 leader_election.cc:288] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163)
I20260807 03:39:16.551546 21011 raft_consensus.cc:482] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:16.573577 20658 raft_consensus.cc:482] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:16.580315 21011 raft_consensus.cc:504] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.552603 20798 raft_consensus.cc:504] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.592871 20960 raft_consensus.cc:686] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:16.592824 19240 leader_election.cc:302] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:16.594156 21011 leader_election.cc:288] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:16.593978 20960 consensus_queue.cc:237] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.594538 20798 leader_election.cc:288] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:16.595352 21035 leader_election.cc:288] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:16.594372 20842 raft_consensus.cc:2781] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:39:16.571964 20912 raft_consensus.cc:482] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
W20260807 03:39:16.596273 20880 consensus_meta.cc:287] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 0.539s user 0.007s sys 0.003s
I20260807 03:39:16.596688 20880 raft_consensus.cc:504] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.596276 20842 raft_consensus.cc:482] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:16.597842 20842 raft_consensus.cc:3037] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:16.596632 20912 raft_consensus.cc:504] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.598702 18073 raft_consensus.cc:2454] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 8.
I20260807 03:39:16.607195 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:16.607445 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:16.607782 18309 raft_consensus.cc:3037] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:16.608268 18073 raft_consensus.cc:3037] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:16.617444 18073 raft_consensus.cc:2454] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:39:16.592690 20691 leader_election.cc:288] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:16.555043 20123 consensus_queue.cc:1046] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
I20260807 03:39:16.618803 18073 raft_consensus.cc:1226] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:16.592729 20658 raft_consensus.cc:3037] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:16.629912 18073 raft_consensus.cc:1261] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 03:39:16.604941 20583 consensus_queue.cc:1035] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20260807 03:39:16.633850 20722 consensus_queue.cc:1035] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
W20260807 03:39:16.636808 21064 log.cc:924] Time spent T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.088s user 0.000s sys 0.000s
I20260807 03:39:16.638711 21013 leader_election.cc:288] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:16.628865 20123 raft_consensus.cc:3032] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Stepping down as leader of term 6
I20260807 03:39:16.640244 20123 raft_consensus.cc:729] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:16.640421 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:16.641206 18189 raft_consensus.cc:2379] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 4: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:16.641331 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:16.642130 18187 raft_consensus.cc:2379] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 5: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:16.632368 21035 raft_consensus.cc:3037] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:16.642076 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:16.642807 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:39:16.627928 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 67473.1us
I20260807 03:39:16.649078 18300 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57276 (request call id 838) took 1131 ms. Trace:
I20260807 03:39:16.644717 20798 raft_consensus.cc:482] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:16.649500 18300 rpcz_store.cc:276] 0807 03:39:15.517170 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:15.517288 (+ 118us) service_pool.cc:227] Handling call
0807 03:39:15.555938 (+ 38650us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:16.448964 (+893026us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:16.648803 (+199839us) spinlock_profiling.cc:243] Waited 876 ms on lock 0x7b54001d1d54. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:39:16.649006 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2014801792}
I20260807 03:39:16.650321 20798 raft_consensus.cc:504] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.650916 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:16.573282 20822 raft_consensus.cc:504] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.653187 20912 leader_election.cc:288] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:16.654181 20822 leader_election.cc:288] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:16.656617 21015 raft_consensus.cc:504] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.660287 18309 raft_consensus.cc:2454] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3.
I20260807 03:39:16.661464 21044 raft_consensus.cc:3032] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Stepping down as leader of term 5
I20260807 03:39:16.661870 21044 raft_consensus.cc:729] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:16.663017 21015 leader_election.cc:288] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:16.662771 21044 consensus_queue.cc:260] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.665359 21059 raft_consensus.cc:504] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.666783 21035 raft_consensus.cc:2726] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:16.666600 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:16.670416 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:16.671308 21048 log.cc:924] Time spent T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.292s user 0.003s sys 0.000s
I20260807 03:39:16.673346 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:16.673413 20658 raft_consensus.cc:504] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.674559 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:16.675447 20658 leader_election.cc:288] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:16.675768 18305 raft_consensus.cc:1226] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 6 LEADER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:16.676995 20694 raft_consensus.cc:504] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.678308 18309 raft_consensus.cc:1226] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:16.679028 20694 leader_election.cc:288] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:16.683223 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:16.685621 18309 raft_consensus.cc:1226] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 7. Current term is 8. Ops: []
I20260807 03:39:16.689186 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:16.667294 21059 leader_election.cc:288] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:16.693303 18300 raft_consensus.cc:2454] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 5.
W20260807 03:39:16.704869 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 50051.1us
I20260807 03:39:16.708652 18300 raft_consensus.cc:1261] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260807 03:39:16.663893 21044 raft_consensus.cc:3037] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:16.709266 18027 leader_election.cc:302] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:16.688347 20583 raft_consensus.cc:482] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:16.709116 20658 consensus_queue.cc:1046] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.033s
I20260807 03:39:16.747344 20658 raft_consensus.cc:3032] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 03:39:16.747700 20658 raft_consensus.cc:729] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
W20260807 03:39:16.747907 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 47189.8us
I20260807 03:39:16.748541 20658 consensus_queue.cc:260] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.749943 20658 raft_consensus.cc:3037] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:16.745689 20583 raft_consensus.cc:504] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.751231 20909 consensus_queue.cc:1035] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:39:16.751905 20583 leader_election.cc:288] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:16.713665 20694 consensus_queue.cc:1046] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20260807 03:39:16.755607 18027 leader_election.cc:302] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:16.735587 18305 raft_consensus.cc:2454] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 8.
I20260807 03:39:16.752516 20653 consensus_queue.cc:1046] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20260807 03:39:16.753285 20123 consensus_queue.cc:260] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:16.759711 18003 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 79e3ecc1bf7d4ca9a7096616efc98585, TS d9716e2bf06442f5b3b0e81087c53c4d -> TS 5c9b5062bdfa46af9396f1a85025a331 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:16.764418 18003 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 79e3ecc1bf7d4ca9a7096616efc98585, TS d9716e2bf06442f5b3b0e81087c53c4d -> TS 5c9b5062bdfa46af9396f1a85025a331 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.766471 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:16.768010 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:16.768777 18312 raft_consensus.cc:2454] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 8.
I20260807 03:39:16.770522 18141 leader_election.cc:302] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:16.770963 20694 raft_consensus.cc:3032] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:39:16.771312 20694 raft_consensus.cc:729] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:16.772056 20694 consensus_queue.cc:260] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.768035 20583 raft_consensus.cc:2781] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election won for term 5
I20260807 03:39:16.773170 20694 raft_consensus.cc:3037] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:16.771343 20999 raft_consensus.cc:2781] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader election won for term 8
I20260807 03:39:16.774106 20999 raft_consensus.cc:686] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 8 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
W20260807 03:39:16.783985 18003 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 6129789267024fac8a337070c10a472a, TS 66a36e1ac8e945e893668e282d394c49 -> TS 5c9b5062bdfa46af9396f1a85025a331 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:16.784428 18003 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 6129789267024fac8a337070c10a472a, TS 66a36e1ac8e945e893668e282d394c49 -> TS 5c9b5062bdfa46af9396f1a85025a331 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.788233 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:16.775185 20999 consensus_queue.cc:237] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.752949 21059 raft_consensus.cc:2781] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:39:16.794410 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
W20260807 03:39:16.822161 18003 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 4cd2b7ebeed04ac6907b51b064e94894, TS d9716e2bf06442f5b3b0e81087c53c4d -> TS 5c9b5062bdfa46af9396f1a85025a331 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:16.822683 18003 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 4cd2b7ebeed04ac6907b51b064e94894, TS d9716e2bf06442f5b3b0e81087c53c4d -> TS 5c9b5062bdfa46af9396f1a85025a331 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.792436 18027 leader_election.cc:302] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:16.752673 20798 leader_election.cc:288] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:16.768568 20880 leader_election.cc:288] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:16.804001 20123 raft_consensus.cc:3037] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:16.837381 21005 raft_consensus.cc:3032] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 LEADER]: Stepping down as leader of term 4
I20260807 03:39:16.837873 21005 raft_consensus.cc:729] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:16.838743 21005 consensus_queue.cc:260] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.773192 20583 raft_consensus.cc:686] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:16.840020 21005 raft_consensus.cc:3037] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:16.840929 20583 consensus_queue.cc:237] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.807065 20842 raft_consensus.cc:504] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:16.810688 21060 consensus_queue.cc:1046] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:39:16.810921 18248 leader_election.cc:302] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:16.811067 20789 consensus_queue.cc:1046] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:39:16.812637 20862 raft_consensus.cc:482] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:16.825729 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
W20260807 03:39:16.851706 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 59418.5us
W20260807 03:39:16.860957 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 846283us
W20260807 03:39:16.862288 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 51447.9us
I20260807 03:39:16.862973 18248 leader_election.cc:302] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:16.864910 18305 raft_consensus.cc:1261] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 03:39:16.866017 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:16.867221 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:16.868711 18073 raft_consensus.cc:1226] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 7. Current term is 8. Ops: []
W20260807 03:39:16.872385 18140 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.15.198.129:36163 timed out after 1.671s (SENT) [suppressed 16 similar messages]
I20260807 03:39:16.877960 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:16.879485 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:16.869971 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:16.883472 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:16.885586 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:16.886834 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:16.891376 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:16.892823 21049 raft_consensus.cc:2726] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 03:39:16.895419 18300 raft_consensus.cc:1261] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 03:39:16.897579 18248 leader_election.cc:302] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:16.898708 18248 leader_election.cc:302] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:16.899726 18248 leader_election.cc:302] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:16.892681 20123 raft_consensus.cc:482] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:16.900910 19292 raft_consensus.cc:1226] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 6 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:16.900687 20123 raft_consensus.cc:504] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.901078 20543 raft_consensus.cc:2726] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260807 03:39:16.901698 20543 raft_consensus.cc:2781] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:39:16.902050 20543 raft_consensus.cc:482] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:16.902105 20900 raft_consensus.cc:2781] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader election won for term 8
I20260807 03:39:16.902428 20543 raft_consensus.cc:3037] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:16.902637 20900 raft_consensus.cc:686] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 8 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:16.903533 20900 consensus_queue.cc:237] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:16.905853 18080 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33268 (request call id 1064) took 1378 ms (client timeout 1601 ms). Trace:
W20260807 03:39:16.906255 18080 rpcz_store.cc:269] 0807 03:39:15.527766 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:15.529059 (+ 1293us) service_pool.cc:227] Handling call
0807 03:39:16.905688 (+1376629us) spinlock_profiling.cc:243] Waited 509 ms on lock 0x7b5400340654. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:16.905800 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1170121984}
I20260807 03:39:16.907435 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:16.907936 20123 leader_election.cc:288] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:16.909432 20511 raft_consensus.cc:3032] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:39:16.909873 20511 raft_consensus.cc:729] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:16.910759 20511 consensus_queue.cc:260] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.858886 20880 raft_consensus.cc:2781] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 03:39:16.911507 20816 consensus_queue.cc:1046] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 03:39:16.912106 20880 raft_consensus.cc:482] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:16.912592 20880 raft_consensus.cc:3037] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:16.912709 20816 raft_consensus.cc:3032] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 03:39:16.913203 20816 raft_consensus.cc:729] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:16.916385 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:16.846370 21059 raft_consensus.cc:686] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:16.922339 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:16.923094 18071 raft_consensus.cc:3037] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:16.926542 20798 raft_consensus.cc:482] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:16.927127 20842 leader_election.cc:288] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 6 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:16.927078 20798 raft_consensus.cc:504] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.874867 20862 raft_consensus.cc:504] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.929127 20798 leader_election.cc:288] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:16.932847 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:16.933543 18080 raft_consensus.cc:2454] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 1.
I20260807 03:39:16.934887 18080 raft_consensus.cc:1261] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 03:39:16.936053 18080 raft_consensus.cc:3037] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 6
I20260807 03:39:16.929665 20123 consensus_queue.cc:1035] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.007s
I20260807 03:39:16.937398 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:16.938197 18305 raft_consensus.cc:2396] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260807 03:39:16.937248 20975 raft_consensus.cc:482] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:16.940071 20975 raft_consensus.cc:504] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.940557 21059 consensus_queue.cc:237] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.942039 20975 leader_election.cc:288] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:16.944057 18310 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57268 (request call id 1130) took 1128 ms. Trace:
I20260807 03:39:16.944545 18310 rpcz_store.cc:276] 0807 03:39:15.815913 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:15.924928 (+109015us) service_pool.cc:227] Handling call
0807 03:39:15.927542 (+ 2614us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:16.311004 (+383462us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:16.943771 (+632767us) spinlock_profiling.cc:243] Waited 362 ms on lock 0x7b5400262ad4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:39:16.943985 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":832241408}
I20260807 03:39:16.946722 18295 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57268 (request call id 1131) took 1128 ms. Trace:
I20260807 03:39:16.947119 18295 rpcz_store.cc:276] 0807 03:39:15.817907 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:15.966439 (+148532us) service_pool.cc:227] Handling call
0807 03:39:15.966913 (+ 474us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:16.529860 (+562947us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:16.946469 (+416609us) spinlock_profiling.cc:243] Waited 560 ms on lock 0x7b54001c3754. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:39:16.946660 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1287517056}
I20260807 03:39:16.948835 20862 leader_election.cc:288] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:16.949412 20511 raft_consensus.cc:3037] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:16.944106 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:16.952589 18300 raft_consensus.cc:2379] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 7: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:16.954514 20123 consensus_queue.cc:1046] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 03:39:16.955317 20653 consensus_queue.cc:1035] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:39:16.956002 20123 raft_consensus.cc:3032] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 7 LEADER]: Stepping down as leader of term 7
I20260807 03:39:16.956904 20123 raft_consensus.cc:729] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:16.954810 18140 leader_election.cc:302] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:16.960551 20994 raft_consensus.cc:2726] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:16.960489 20123 consensus_queue.cc:260] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.961845 20123 raft_consensus.cc:3037] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:16.962081 18140 leader_election.cc:302] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:16.962862 20994 raft_consensus.cc:2781] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:39:16.963217 20994 raft_consensus.cc:482] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:16.963573 20994 raft_consensus.cc:3037] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:16.968595 19617 raft_consensus.cc:3032] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 03:39:16.968807 18140 leader_election.cc:302] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:16.968976 19617 raft_consensus.cc:729] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:16.969542 20653 raft_consensus.cc:2781] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:39:16.969900 20653 raft_consensus.cc:482] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:16.969735 19617 consensus_queue.cc:260] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.971041 19617 raft_consensus.cc:3037] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:16.920504 20765 consensus_queue.cc:1035] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.079s
W20260807 03:39:16.973376 21086 log.cc:924] Time spent T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
W20260807 03:39:16.975945 18246 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.132:41381, user_credentials={real_user=slave}} blocked reactor thread for 44156.7us
I20260807 03:39:16.978431 20888 raft_consensus.cc:3032] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 LEADER]: Stepping down as leader of term 7
I20260807 03:39:16.978760 20888 raft_consensus.cc:729] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:16.981313 18297 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57268 (request call id 1105) took 2215 ms. Trace:
W20260807 03:39:16.982681 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 83012.8us
I20260807 03:39:16.981690 18297 rpcz_store.cc:276] 0807 03:39:14.766147 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:14.767160 (+ 1013us) service_pool.cc:227] Handling call
0807 03:39:14.767549 (+ 389us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 03:39:14.923901 (+156352us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:14.923939 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 03:39:14.944442 (+ 20503us) log.cc:842] Serialized 52 byte log entry
0807 03:39:15.763208 (+818766us) raft_consensus.cc:1631] Marking committed up to 2
0807 03:39:15.777871 (+ 14663us) log.cc:842] Serialized 14 byte log entry
0807 03:39:16.146909 (+369038us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:16.801569 (+654660us) spinlock_profiling.cc:243] Waited 10.8 ms on lock 0x7b180041a3a0. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe9981368b1 00007fe998143661 00007fe98f297638 00007fe98f29ab11 00007fe98f29aac9 00007fe98f29aa59 00007fe98f29aa21 00007fe98f299d1d 00007fe9984fb204 00007fe9984fb039 00007fe98f279dc6
0807 03:39:16.980984 (+179415us) spinlock_profiling.cc:243] Waited 152 ms on lock 0x7b54002634d4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:39:16.981036 (+ 52us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:16.981100 (+ 64us) raft_consensus.cc:1671] finished
0807 03:39:16.981114 (+ 14us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:16.981258 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9026,"prepare.queue_time_us":14428,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":38400,"raft.run_cpu_time_us":267,"raft.run_wall_time_us":22206,"spinlock_wait_cycles":375364736,"thread_start_us":1124847,"threads_started":2,"wal-append.queue_time_us":892869}
I20260807 03:39:16.983697 20960 raft_consensus.cc:2781] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:39:16.984272 20960 raft_consensus.cc:686] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:16.985616 20789 consensus_queue.cc:1046] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:39:16.985370 20960 consensus_queue.cc:237] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.987310 21013 raft_consensus.cc:3032] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:39:16.987789 21013 raft_consensus.cc:729] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:16.988883 20822 raft_consensus.cc:3032] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 03:39:16.989281 20822 raft_consensus.cc:729] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:16.989454 18311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57268 (request call id 1122) took 1188 ms. Trace:
I20260807 03:39:16.989948 18311 rpcz_store.cc:276] 0807 03:39:15.801323 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:15.801468 (+ 145us) service_pool.cc:227] Handling call
0807 03:39:15.804872 (+ 3404us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:16.528804 (+723932us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:16.989186 (+460382us) spinlock_profiling.cc:243] Waited 718 ms on lock 0x7b54002b4dd4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:39:16.989384 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1652341632}
I20260807 03:39:16.990064 20822 consensus_queue.cc:260] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.991232 20822 raft_consensus.cc:3037] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:16.991451 21043 raft_consensus.cc:3032] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:39:16.991923 21043 raft_consensus.cc:729] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:16.992834 21043 consensus_queue.cc:260] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:16.994648 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 73033.9us
I20260807 03:39:16.994830 18184 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44660 (request call id 1348) took 1264 ms. Trace:
I20260807 03:39:16.995286 18184 rpcz_store.cc:276] 0807 03:39:15.754644 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:15.901855 (+147211us) service_pool.cc:227] Handling call
0807 03:39:16.590707 (+688852us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:16.590960 (+ 253us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:16.590999 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:16.591048 (+ 49us) raft_consensus.cc:1631] Marking committed up to 5
0807 03:39:16.591090 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:16.591159 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:16.994650 (+403491us) spinlock_profiling.cc:243] Waited 688 ms on lock 0x7b54003c9450. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bea09 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:16.994771 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1583533952}
I20260807 03:39:16.970259 20653 raft_consensus.cc:3037] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
W20260807 03:39:17.000535 18003 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 96140f11b8e74293b333cd7e760c3959, TS 66a36e1ac8e945e893668e282d394c49 -> TS 5c9b5062bdfa46af9396f1a85025a331 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:16.999037 18140 leader_election.cc:302] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
W20260807 03:39:17.009202 18003 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 96140f11b8e74293b333cd7e760c3959, TS 66a36e1ac8e945e893668e282d394c49 -> TS 5c9b5062bdfa46af9396f1a85025a331 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:17.010545 18071 raft_consensus.cc:2454] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 6.
I20260807 03:39:17.015616 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:17.040398 18073 raft_consensus.cc:2396] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260807 03:39:17.043520 18140 leader_election.cc:302] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:17.046850 18300 raft_consensus.cc:1226] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 7. Current term is 8. Ops: []
I20260807 03:39:16.914116 20816 consensus_queue.cc:260] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
W20260807 03:39:17.074787 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 67537.1us
I20260807 03:39:17.080674 18248 leader_election.cc:302] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:17.037983 20658 raft_consensus.cc:2781] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:39:16.979532 20888 consensus_queue.cc:260] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:17.042164 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:17.109987 18311 raft_consensus.cc:2379] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 3: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:17.036636 20746 raft_consensus.cc:482] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:17.034066 21015 consensus_queue.cc:1046] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.074s
I20260807 03:39:17.037801 20909 consensus_queue.cc:1046] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.074s
I20260807 03:39:17.036619 20882 consensus_queue.cc:1035] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.077s
I20260807 03:39:17.039033 20813 consensus_queue.cc:1046] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.081s
I20260807 03:39:17.114004 20678 raft_consensus.cc:2781] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260807 03:39:17.122541 21076 log.cc:924] Time spent T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20260807 03:39:17.117403 18080 raft_consensus.cc:1261] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260807 03:39:17.141217 18193 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
mode: GRACEFUL
new_leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:34758
I20260807 03:39:17.145183 20880 raft_consensus.cc:504] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:17.118289 21020 log.cc:924] Time spent T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
I20260807 03:39:17.107187 20658 raft_consensus.cc:482] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:17.166929 18193 raft_consensus.cc:595] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Received request to transfer leadership to TS d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:17.167254 20658 raft_consensus.cc:3037] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:17.167392 18193 raft_consensus.cc:603] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 03:39:16.988905 21013 consensus_queue.cc:260] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:17.169118 21013 raft_consensus.cc:3037] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:17.171386 18081 raft_consensus.cc:2354] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 8. Current term is 9.
I20260807 03:39:17.122388 20862 raft_consensus.cc:482] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:17.172435 20862 raft_consensus.cc:504] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:17.173179 20880 leader_election.cc:288] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:17.174790 20862 leader_election.cc:288] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:17.081857 21057 consensus_queue.cc:1035] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.059s
I20260807 03:39:17.132613 20888 raft_consensus.cc:3037] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Advancing to term 8
W20260807 03:39:17.179795 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 143712us
W20260807 03:39:17.112497 20620 consensus_queue.cc:1159] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "3c4bceb836db4536a0f255fdf86abc02" responder_term: 5 status { last_received { term: 5 index: 5 } last_committed_idx: 5 last_received_current_leader { term: 5 index: 5 } }
W20260807 03:39:17.183132 21093 log.cc:924] Time spent T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
W20260807 03:39:17.212698 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 58044us
I20260807 03:39:17.157385 20678 raft_consensus.cc:482] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:17.130231 20746 raft_consensus.cc:504] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:17.178777 20882 raft_consensus.cc:3032] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 03:39:17.201385 20862 raft_consensus.cc:482] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:17.114442 20999 raft_consensus.cc:2726] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:17.219702 20678 raft_consensus.cc:3037] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:17.237125 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:17.237941 18300 raft_consensus.cc:3037] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:17.245416 18180 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "60437eed8852484c88e84993e376670e"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
mode: GRACEFUL
new_leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:34758
I20260807 03:39:17.219384 20882 raft_consensus.cc:729] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:17.247356 18193 raft_consensus.cc:1261] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 03:39:17.256798 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:17.257254 18297 raft_consensus.cc:1261] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 03:39:17.259919 18300 raft_consensus.cc:2454] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 9.
I20260807 03:39:17.265992 18027 leader_election.cc:302] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:17.269510 18310 raft_consensus.cc:1261] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260807 03:39:17.234813 20994 raft_consensus.cc:504] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:17.271504 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:17.273156 20994 leader_election.cc:288] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:17.275372 19240 leader_election.cc:302] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:17.276309 21063 raft_consensus.cc:2726] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260807 03:39:17.279753 21098 log.cc:924] Time spent T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20260807 03:39:17.239020 20800 raft_consensus.cc:482] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:17.281314 20511 consensus_queue.cc:1035] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:39:17.284332 21005 raft_consensus.cc:2781] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader election won for term 9
W20260807 03:39:17.284839 21100 log.cc:924] Time spent T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.211s user 0.003s sys 0.000s
I20260807 03:39:17.285030 18184 raft_consensus.cc:1226] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:17.284865 21005 raft_consensus.cc:686] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 9 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:17.288594 21005 consensus_queue.cc:237] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:17.231648 20862 raft_consensus.cc:504] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:17.280694 20800 raft_consensus.cc:504] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:17.314179 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:17.291801 20543 raft_consensus.cc:504] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:17.296407 20658 raft_consensus.cc:504] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:17.307410 20999 consensus_queue.cc:1035] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20260807 03:39:17.316927 20721 consensus_queue.cc:1046] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.026s
I20260807 03:39:17.305783 21108 consensus_queue.cc:1035] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.043s
I20260807 03:39:17.355183 20746 leader_election.cc:288] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:17.316262 20882 consensus_queue.cc:260] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:17.368659 20862 leader_election.cc:288] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:17.435245 18180 raft_consensus.cc:595] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Received request to transfer leadership to TS d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:17.436458 18180 raft_consensus.cc:603] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 03:39:17.460948 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:17.462625 18297 raft_consensus.cc:2454] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 6.
I20260807 03:39:17.464344 18027 leader_election.cc:302] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:17.466264 18298 raft_consensus.cc:1226] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3. Ops: []
I20260807 03:39:17.473968 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:17.479907 20543 leader_election.cc:288] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:17.484853 20653 raft_consensus.cc:504] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:17.489068 20123 consensus_meta.cc:287] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 0.527s user 0.018s sys 0.000s
I20260807 03:39:17.490944 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:17.491402 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:17.492658 18297 raft_consensus.cc:2354] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 6. Current term is 7.
I20260807 03:39:17.493219 18312 raft_consensus.cc:1261] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 03:39:17.494642 18141 leader_election.cc:398] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Vote denied by peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021) with higher term. Message: Invalid argument: T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 6. Current term is 7.
I20260807 03:39:17.494997 18141 leader_election.cc:401] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260807 03:39:17.501137 20651 raft_consensus.cc:3037] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:17.501418 18297 raft_consensus.cc:1226] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:17.504582 20658 leader_election.cc:288] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:17.514042 20651 raft_consensus.cc:2726] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021) with higher term. Message: Invalid argument: T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 6. Current term is 7.
I20260807 03:39:17.518913 20798 raft_consensus.cc:2781] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:39:17.524621 20882 raft_consensus.cc:3037] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:17.521134 20800 leader_election.cc:288] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:17.521171 20804 consensus_queue.cc:1035] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 03:39:17.524422 20798 raft_consensus.cc:482] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:17.528234 20798 raft_consensus.cc:3037] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Advancing to term 7
W20260807 03:39:17.527904 20822 consensus_meta.cc:287] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.536s user 0.019s sys 0.002s
I20260807 03:39:17.534332 20124 consensus_queue.cc:1046] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 03:39:17.537312 20798 raft_consensus.cc:504] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:17.556598 20798 leader_election.cc:288] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:17.571029 21015 raft_consensus.cc:3032] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 03:39:17.571699 21015 raft_consensus.cc:729] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:17.573194 21015 consensus_queue.cc:260] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:17.574679 21015 raft_consensus.cc:3037] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:17.577658 20277 raft_consensus.cc:3032] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 03:39:17.578064 20277 raft_consensus.cc:729] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:17.579260 20277 consensus_queue.cc:260] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:17.588157 20277 raft_consensus.cc:3037] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:17.582607 20917 raft_consensus.cc:3032] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 03:39:17.588923 20917 raft_consensus.cc:729] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:17.590474 20917 consensus_queue.cc:260] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:17.598466 18312 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585"
dest_uuid: "66a36e1ac8e945e893668e282d394c49"
mode: GRACEFUL
new_leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:58736
I20260807 03:39:17.599383 18312 raft_consensus.cc:595] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Received request to transfer leadership to TS d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:17.599828 18312 raft_consensus.cc:603] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 03:39:17.601050 20678 raft_consensus.cc:504] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:17.598948 20816 raft_consensus.cc:3037] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:17.603472 20678 leader_election.cc:288] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:17.608609 18312 raft_consensus.cc:1261] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:39:17.607906 18180 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
mode: GRACEFUL
new_leader_uuid: "5c9b5062bdfa46af9396f1a85025a331"
from {username='slave'} at 127.0.0.1:34758
I20260807 03:39:17.616835 18180 raft_consensus.cc:595] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Received request to transfer leadership to TS 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:39:17.617434 18180 raft_consensus.cc:603] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 03:39:17.617620 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:17.619185 21057 raft_consensus.cc:482] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5c9b5062bdfa46af9396f1a85025a331)
I20260807 03:39:17.620993 18180 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
mode: GRACEFUL
new_leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:34758
I20260807 03:39:17.621714 18180 raft_consensus.cc:595] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Received request to transfer leadership to TS d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:17.622187 18180 raft_consensus.cc:603] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 03:39:17.624681 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:17.625103 18193 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
mode: GRACEFUL
new_leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:34758
I20260807 03:39:17.625695 18193 raft_consensus.cc:595] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Received request to transfer leadership to TS d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:17.626204 18193 raft_consensus.cc:603] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 03:39:17.626258 20917 raft_consensus.cc:3037] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:17.627476 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260807 03:39:17.628561 18293 raft_consensus.cc:1226] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:17.637820 20912 consensus_queue.cc:1046] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 03:39:17.619972 21057 raft_consensus.cc:504] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:17.590991 20653 leader_election.cc:288] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:17.642683 18140 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.598s user 0.015s sys 0.044s
W20260807 03:39:17.645665 18140 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.601s user 0.015s sys 0.044s
I20260807 03:39:17.648332 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
W20260807 03:39:17.649474 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 605973us
I20260807 03:39:17.650465 18184 raft_consensus.cc:1226] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 1. Current term is 2. Ops: []
I20260807 03:39:17.650976 18140 leader_election.cc:302] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:17.652125 20912 raft_consensus.cc:2726] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:17.652894 20912 raft_consensus.cc:3032] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 03:39:17.653273 20912 raft_consensus.cc:729] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:17.638438 20651 consensus_queue.cc:1046] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
I20260807 03:39:17.654649 20651 raft_consensus.cc:3032] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 03:39:17.655218 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:39:17.655366 20651 raft_consensus.cc:729] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:17.656603 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:39:17.658834 18180 raft_consensus.cc:1226] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 2. Current term is 3. Ops: []
I20260807 03:39:17.659274 18140 leader_election.cc:302] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:17.660328 21115 raft_consensus.cc:2726] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:17.660403 21043 raft_consensus.cc:3037] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:17.662096 18140 leader_election.cc:302] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:17.662312 21057 leader_election.cc:288] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:17.664034 21115 raft_consensus.cc:3037] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:17.654361 20912 consensus_queue.cc:260] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:17.647238 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:17.671221 18297 raft_consensus.cc:3037] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:17.685310 19292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:17.687621 20583 raft_consensus.cc:482] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:17.694239 21060 consensus_queue.cc:1046] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260807 03:39:17.697821 17915 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35098 (request call id 25) took 6816 ms. Trace:
I20260807 03:39:17.698385 20583 raft_consensus.cc:504] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:17.693799 18180 raft_consensus.cc:1261] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
W20260807 03:39:17.707851 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 45948.4us
I20260807 03:39:17.710271 20694 raft_consensus.cc:482] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:17.710804 21060 raft_consensus.cc:3032] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:39:17.711319 21060 raft_consensus.cc:729] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:17.710974 20694 raft_consensus.cc:504] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:17.712598 21060 consensus_queue.cc:260] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:17.714064 21060 raft_consensus.cc:3037] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:17.706175 18180 pending_rounds.cc:76] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02: Aborting all ops after (but not including) 1
I20260807 03:39:17.704491 20912 raft_consensus.cc:3037] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:17.656626 20651 consensus_queue.cc:260] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:17.718593 18180 pending_rounds.cc:97] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260807 03:39:17.723064 18180 raft_consensus.cc:2866] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 03:39:17.726579 18297 raft_consensus.cc:2454] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 7.
I20260807 03:39:17.726987 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:17.728080 21035 consensus_queue.cc:1046] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 03:39:17.728587 20651 raft_consensus.cc:3037] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:17.704715 17915 rpcz_store.cc:276] 0807 03:39:10.881589 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:10.885703 (+ 4114us) service_pool.cc:227] Handling call
0807 03:39:17.697496 (+6811793us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":391905,"reported_tablets":3}
I20260807 03:39:17.730861 20583 leader_election.cc:288] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:17.731957 21061 consensus_queue.cc:1035] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20260807 03:39:17.734928 21115 raft_consensus.cc:2726] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:17.735615 18140 leader_election.cc:302] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
W20260807 03:39:17.728130 18248 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.570s user 0.003s sys 0.031s
I20260807 03:39:17.714092 20694 leader_election.cc:288] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:17.747468 18140 leader_election.cc:302] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:17.744626 21061 raft_consensus.cc:482] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:17.749238 21057 consensus_queue.cc:1035] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 03:39:17.749812 20678 raft_consensus.cc:3032] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 03:39:17.752161 21115 raft_consensus.cc:2781] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:39:17.766366 20678 raft_consensus.cc:729] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:17.766659 21115 raft_consensus.cc:482] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
W20260807 03:39:17.758374 18248 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.650s user 0.003s sys 0.032s
I20260807 03:39:17.761370 17920 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39492 (request call id 146) took 7253 ms. Trace:
I20260807 03:39:17.767230 21115 raft_consensus.cc:3037] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:17.768953 21061 raft_consensus.cc:504] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:17.768973 20678 consensus_queue.cc:260] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:17.775512 18027 leader_election.cc:302] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:17.774648 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:17.777634 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 665067us
I20260807 03:39:17.777933 20678 raft_consensus.cc:3037] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
W20260807 03:39:17.791819 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 44507.5us
I20260807 03:39:17.799088 21061 leader_election.cc:288] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:17.795732 18140 leader_election.cc:302] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:17.801801 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:17.780519 20658 raft_consensus.cc:2726] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 03:39:17.767671 17920 rpcz_store.cc:276] 0807 03:39:10.508353 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:10.514581 (+ 6228us) service_pool.cc:227] Handling call
0807 03:39:17.761181 (+7246600us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 03:39:15.903294 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 03:39:15.998734 (+ 95440us) write_op.cc:432] Acquiring schema lock in shared mode
0807 03:39:16.011336 (+ 12602us) write_op.cc:435] Acquired schema lock
0807 03:39:16.011364 (+ 28us) tablet.cc:660] Decoding operations
0807 03:39:16.302989 (+291625us) write_op.cc:620] Acquiring the partition lock for write op
0807 03:39:16.316153 (+ 13164us) write_op.cc:641] Partition lock acquired for write op
0807 03:39:16.319588 (+ 3435us) tablet.cc:683] Acquiring locks for 17 operations
0807 03:39:16.421126 (+101538us) tablet.cc:699] Row locks acquired
0807 03:39:16.421149 (+ 23us) write_op.cc:260] PREPARE: finished
0807 03:39:16.421336 (+ 187us) write_op.cc:270] Start()
0807 03:39:16.421454 (+ 118us) write_op.cc:276] Timestamp: P: 1786073956421280 usec, L: 0
0807 03:39:16.421477 (+ 23us) op_driver.cc:331] REPLICATION: starting
0807 03:39:16.478804 (+ 57327us) log.cc:842] Serialized 6786 byte log entry
0807 03:39:17.268346 (+789542us) op_driver.cc:444] REPLICATION: finished
0807 03:39:17.383424 (+115078us) write_op.cc:301] APPLY: starting
0807 03:39:17.383540 (+ 116us) tablet.cc:1365] starting BulkCheckPresence
0807 03:39:17.383988 (+ 448us) tablet.cc:1368] finished BulkCheckPresence
0807 03:39:17.384159 (+ 171us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 03:39:17.386202 (+ 2043us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 03:39:17.386567 (+ 365us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=17
0807 03:39:17.386651 (+ 84us) write_op.cc:312] APPLY: finished
0807 03:39:17.387126 (+ 475us) log.cc:842] Serialized 118 byte log entry
0807 03:39:17.387333 (+ 207us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:39:17.387561 (+ 228us) write_op.cc:454] Released schema lock
0807 03:39:17.387631 (+ 70us) write_op.cc:341] FINISH: Updating metrics
0807 03:39:17.387930 (+ 299us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:39:17.387981 (+ 51us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":509263,"reported_tablets":36,"child_traces":[["op",{"apply.queue_time_us":114744,"apply.run_cpu_time_us":5134,"apply.run_wall_time_us":5135,"num_ops":17,"prepare.queue_time_us":310010,"prepare.run_cpu_time_us":21184,"prepare.run_wall_time_us":763729,"raft.queue_time_us":501086,"raft.run_cpu_time_us":9906,"raft.run_wall_time_us":120785,"replication_time_us":846781,"thread_start_us":894023,"threads_started":4,"wal-append.queue_time_us":248186}]]}
I20260807 03:39:17.785125 21043 raft_consensus.cc:2781] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election won for term 7
I20260807 03:39:17.813441 21043 raft_consensus.cc:686] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:17.819533 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:17.820057 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:17.821192 18069 raft_consensus.cc:3037] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:17.825093 18080 raft_consensus.cc:1226] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260807 03:39:17.826545 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:17.827260 18073 raft_consensus.cc:3037] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:17.829981 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:17.830996 18071 raft_consensus.cc:2379] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 6: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:17.832212 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:17.832926 18072 raft_consensus.cc:2454] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:39:17.833913 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:17.834568 18079 raft_consensus.cc:2454] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
I20260807 03:39:17.839054 18069 raft_consensus.cc:2454] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 7.
I20260807 03:39:17.844596 18073 raft_consensus.cc:2454] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:39:17.814244 20658 raft_consensus.cc:2726] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 03:39:17.848433 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:17.850294 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 54692.1us
I20260807 03:39:17.854486 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:17.869374 18081 raft_consensus.cc:2454] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
I20260807 03:39:17.870427 18071 raft_consensus.cc:1226] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260807 03:39:17.852032 21059 raft_consensus.cc:482] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:17.871613 21059 raft_consensus.cc:504] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:17.873759 21059 leader_election.cc:288] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:17.879554 21061 raft_consensus.cc:482] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:17.880221 21061 raft_consensus.cc:504] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:17.882973 20583 raft_consensus.cc:482] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:17.883843 21061 leader_election.cc:288] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:17.884052 19292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:17.883566 20583 raft_consensus.cc:504] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:17.886525 20583 leader_election.cc:288] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:17.884368 21061 raft_consensus.cc:482] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:17.896270 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:17.896873 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:17.896945 18298 raft_consensus.cc:2454] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 6.
I20260807 03:39:17.898514 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:17.899107 18309 raft_consensus.cc:2454] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 2.
I20260807 03:39:17.903426 21059 raft_consensus.cc:482] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:17.903990 21059 raft_consensus.cc:504] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:17.905941 21059 leader_election.cc:288] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:17.908001 18309 raft_consensus.cc:1261] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260807 03:39:17.912240 18069 raft_consensus.cc:2454] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:17.921594 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:17.922199 18071 raft_consensus.cc:1226] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:17.911909 21043 consensus_queue.cc:237] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:17.916262 18140 leader_election.cc:302] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:17.921701 18027 leader_election.cc:302] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:17.924703 20798 raft_consensus.cc:2781] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:39:17.925084 20658 raft_consensus.cc:2726] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:17.925112 20798 raft_consensus.cc:482] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:17.925830 20798 raft_consensus.cc:3037] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:17.926007 21059 consensus_queue.cc:1035] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 03:39:17.927047 18140 leader_election.cc:302] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:17.913977 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:17.928254 20658 raft_consensus.cc:2726] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:17.930496 18248 leader_election.cc:302] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:17.930379 20277 consensus_queue.cc:1046] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:39:17.931264 21061 raft_consensus.cc:504] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:17.933468 18309 raft_consensus.cc:2454] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 6.
I20260807 03:39:17.934499 20798 raft_consensus.cc:504] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:17.937569 21115 raft_consensus.cc:504] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:17.937227 18248 leader_election.cc:302] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:17.937489 21060 raft_consensus.cc:2726] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 03:39:17.936549 21059 raft_consensus.cc:482] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:17.942871 21122 raft_consensus.cc:3032] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:39:17.943564 21057 raft_consensus.cc:2781] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:17.943200 21059 raft_consensus.cc:504] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:17.943859 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:17.942092 18079 raft_consensus.cc:1226] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 1. Current term is 2. Ops: []
I20260807 03:39:17.949183 21115 leader_election.cc:288] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:17.950309 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:17.952971 21059 leader_election.cc:288] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:17.953184 20583 raft_consensus.cc:482] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:17.943966 21057 raft_consensus.cc:482] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:17.953670 21057 raft_consensus.cc:3037] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:17.953743 20583 raft_consensus.cc:504] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:17.954234 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:17.955737 20583 leader_election.cc:288] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:17.943302 21122 raft_consensus.cc:729] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:17.948493 18248 leader_election.cc:302] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:17.957121 21122 consensus_queue.cc:260] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:17.958835 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:17.959101 20678 raft_consensus.cc:2781] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:17.959453 18298 raft_consensus.cc:2454] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 4.
I20260807 03:39:17.960206 18248 leader_election.cc:302] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:17.959478 20678 raft_consensus.cc:482] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:17.961098 20277 raft_consensus.cc:2781] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 03:39:17.961665 18140 leader_election.cc:302] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:17.964959 18027 leader_election.cc:302] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:17.964315 20277 raft_consensus.cc:686] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:17.958842 21122 raft_consensus.cc:3037] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:17.969435 20277 consensus_queue.cc:237] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:17.972237 21061 leader_election.cc:288] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:17.964627 20678 raft_consensus.cc:3037] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:17.968060 21043 raft_consensus.cc:2781] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:17.942305 20798 leader_election.cc:288] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:17.977556 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:17.974653 21043 raft_consensus.cc:482] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:17.981037 21043 raft_consensus.cc:3037] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:17.978155 19292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:17.984087 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:17.984946 18071 raft_consensus.cc:3037] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:17.977993 21115 raft_consensus.cc:2781] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:17.989614 21115 raft_consensus.cc:482] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:17.989984 21115 raft_consensus.cc:3037] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:17.992630 18027 leader_election.cc:302] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:17.993389 20583 raft_consensus.cc:482] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:17.993701 20798 raft_consensus.cc:2726] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:17.993934 20583 raft_consensus.cc:504] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:17.995977 20583 leader_election.cc:288] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:17.964058 21057 raft_consensus.cc:504] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.003609 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:18.000698 18309 raft_consensus.cc:2454] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 2.
I20260807 03:39:18.008936 18027 leader_election.cc:302] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:18.012737 21061 raft_consensus.cc:2781] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:39:18.022287 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:18.030287 21057 leader_election.cc:288] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163)
I20260807 03:39:18.031548 18071 raft_consensus.cc:2454] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:39:18.033818 18079 raft_consensus.cc:3037] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:18.034046 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:18.040302 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:18.044565 18081 raft_consensus.cc:1261] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:39:18.022353 21043 raft_consensus.cc:504] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.051741 18140 leader_election.cc:302] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:18.036259 21061 raft_consensus.cc:686] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:18.054883 20658 raft_consensus.cc:3037] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:18.035089 20678 raft_consensus.cc:504] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.071332 18140 leader_election.cc:302] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:18.073871 20917 raft_consensus.cc:2781] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:18.074348 18081 raft_consensus.cc:1261] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 03:39:18.074288 20917 raft_consensus.cc:482] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:18.079205 20917 raft_consensus.cc:3037] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:18.079720 21043 leader_election.cc:288] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:18.081910 20678 leader_election.cc:288] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:18.080879 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:18.100656 18298 raft_consensus.cc:3037] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:18.084244 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:18.085048 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:18.085903 18248 leader_election.cc:302] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:18.091502 18079 raft_consensus.cc:2454] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 6.
I20260807 03:39:18.101294 18081 raft_consensus.cc:3037] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:18.102074 18180 raft_consensus.cc:2379] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 4: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:18.091825 20651 raft_consensus.cc:482] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:18.094309 20917 raft_consensus.cc:504] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.096616 18140 leader_election.cc:302] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:18.103024 18248 leader_election.cc:302] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:18.102949 20822 raft_consensus.cc:2781] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:39:18.118269 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:18.125505 20653 raft_consensus.cc:2726] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:18.125578 20912 raft_consensus.cc:482] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:18.129346 20912 raft_consensus.cc:504] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.105439 21115 raft_consensus.cc:504] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.134037 18140 leader_election.cc:302] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:18.121513 20798 raft_consensus.cc:482] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:18.135426 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:18.113018 21061 consensus_queue.cc:237] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.128854 20678 raft_consensus.cc:2781] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 03:39:18.147040 20653 raft_consensus.cc:2726] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:18.147241 20678 raft_consensus.cc:686] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:18.148135 20912 leader_election.cc:288] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:18.121774 20651 raft_consensus.cc:504] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:18.148288 20678 consensus_queue.cc:237] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.150904 20651 leader_election.cc:288] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:18.130223 20822 raft_consensus.cc:686] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:18.152967 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:18.153895 18081 raft_consensus.cc:2454] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
I20260807 03:39:18.144613 18079 raft_consensus.cc:2454] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
I20260807 03:39:18.153607 18293 raft_consensus.cc:2454] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 8.
I20260807 03:39:18.137302 20917 leader_election.cc:288] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:18.156972 18248 leader_election.cc:302] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:18.158886 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:18.159610 18141 leader_election.cc:302] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:18.159966 18293 raft_consensus.cc:2379] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 6: Already voted for candidate 5c9b5062bdfa46af9396f1a85025a331 in this term.
I20260807 03:39:18.148186 20658 raft_consensus.cc:2726] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:18.152066 19292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:18.144856 20798 raft_consensus.cc:504] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.161010 20277 raft_consensus.cc:2726] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 03:39:18.161296 19292 raft_consensus.cc:2379] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 6: Already voted for candidate 5c9b5062bdfa46af9396f1a85025a331 in this term.
I20260807 03:39:18.162585 20798 leader_election.cc:288] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:18.164361 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:18.165095 18309 raft_consensus.cc:2379] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 8: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:18.167636 21108 raft_consensus.cc:482] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:18.174548 18141 leader_election.cc:302] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:18.180774 18140 leader_election.cc:302] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:18.183563 18180 raft_consensus.cc:1226] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:18.185638 21056 consensus_queue.cc:1046] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 03:39:18.172268 20658 raft_consensus.cc:2781] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 03:39:18.187026 20917 raft_consensus.cc:3037] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:18.186589 21056 raft_consensus.cc:3032] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 03:39:18.188674 21056 raft_consensus.cc:729] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:18.151979 20822 consensus_queue.cc:237] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.176151 20798 raft_consensus.cc:482] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:18.168206 21108 raft_consensus.cc:504] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.191730 20651 raft_consensus.cc:2726] T af8877c3fbba4ab18765832f1390921f P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:18.192041 18140 leader_election.cc:302] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:18.192559 21056 consensus_queue.cc:260] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.187963 18248 leader_election.cc:302] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:18.195508 18298 raft_consensus.cc:2454] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 5.
I20260807 03:39:18.189205 20658 raft_consensus.cc:482] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:18.203228 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:18.205546 18027 leader_election.cc:302] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:18.206351 21043 raft_consensus.cc:2781] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election won for term 5
I20260807 03:39:18.206823 21043 raft_consensus.cc:686] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:18.207667 21043 consensus_queue.cc:237] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.201660 20678 raft_consensus.cc:2781] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:18.199095 20658 raft_consensus.cc:3037] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:18.212762 20678 raft_consensus.cc:482] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:18.221240 20678 raft_consensus.cc:3037] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:18.199311 20912 raft_consensus.cc:2726] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:18.214072 20917 raft_consensus.cc:2726] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:18.217126 21108 leader_election.cc:288] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:18.218981 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:18.214885 18140 leader_election.cc:302] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:18.203521 21115 leader_election.cc:288] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:18.241668 18293 raft_consensus.cc:2454] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 6.
I20260807 03:39:18.241226 21061 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:18.245141 21061 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.247519 21061 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:18.201972 20798 raft_consensus.cc:504] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.229249 21056 raft_consensus.cc:3037] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:18.248190 18141 leader_election.cc:302] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:18.250516 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:18.252365 21115 raft_consensus.cc:2781] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:39:18.252455 20277 raft_consensus.cc:482] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:18.253304 18248 leader_election.cc:302] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:18.253036 20277 raft_consensus.cc:504] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.254663 21035 raft_consensus.cc:2781] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:39:18.254740 18248 leader_election.cc:302] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:18.255211 21035 raft_consensus.cc:686] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:18.255952 20277 leader_election.cc:288] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:18.256243 20816 raft_consensus.cc:2726] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:18.258306 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:39:18.258445 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 43708.5us
I20260807 03:39:18.256278 20854 consensus_queue.cc:1035] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260807 03:39:18.259383 18140 leader_election.cc:302] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:18.260380 20651 raft_consensus.cc:2726] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 03:39:18.252789 21115 raft_consensus.cc:482] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:18.261229 18140 leader_election.cc:302] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:18.261454 21115 raft_consensus.cc:3037] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:18.262799 20653 raft_consensus.cc:2781] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:39:18.262923 18140 leader_election.cc:302] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:18.263196 20653 raft_consensus.cc:482] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:18.263615 20653 raft_consensus.cc:3037] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:18.263455 20651 consensus_queue.cc:1035] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 03:39:18.251219 20912 raft_consensus.cc:2726] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:18.264267 21015 raft_consensus.cc:2781] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 03:39:18.266127 21015 raft_consensus.cc:686] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:18.266965 21015 consensus_queue.cc:237] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.269047 20816 consensus_queue.cc:1035] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:39:18.270223 20798 leader_election.cc:288] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:18.243400 20917 raft_consensus.cc:482] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:18.270928 20917 raft_consensus.cc:504] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.272819 20917 leader_election.cc:288] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:18.276755 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:18.277382 18298 raft_consensus.cc:2454] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 7.
I20260807 03:39:18.278399 18141 leader_election.cc:302] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:18.257854 21035 consensus_queue.cc:237] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.288682 20917 raft_consensus.cc:2781] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 03:39:18.298655 20917 raft_consensus.cc:482] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:18.298995 20917 raft_consensus.cc:3037] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:18.286355 20653 raft_consensus.cc:504] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.302400 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:18.283089 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:18.304759 18298 raft_consensus.cc:2454] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 6.
I20260807 03:39:18.305907 18027 leader_election.cc:302] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:18.306294 20658 raft_consensus.cc:504] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.306911 21061 raft_consensus.cc:2781] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:39:18.307391 21061 raft_consensus.cc:482] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:18.307802 21061 raft_consensus.cc:3037] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:18.308714 20912 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:18.309329 20912 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.311689 20912 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:18.312876 20678 raft_consensus.cc:504] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.314117 18298 raft_consensus.cc:1226] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:18.315124 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:18.315544 20678 leader_election.cc:288] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:18.315727 20653 leader_election.cc:288] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:18.318310 21108 consensus_queue.cc:1046] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:39:18.320978 21108 raft_consensus.cc:3032] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:39:18.321385 21108 raft_consensus.cc:729] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:18.319614 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:18.322528 18180 raft_consensus.cc:2379] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 5: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:18.324365 21108 consensus_queue.cc:260] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.326365 21108 raft_consensus.cc:3037] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:18.328235 18140 leader_election.cc:302] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:18.329030 20653 raft_consensus.cc:3037] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
W20260807 03:39:18.329100 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 64987.9us
I20260807 03:39:18.329860 18140 leader_election.cc:302] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:18.331255 20912 raft_consensus.cc:2781] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 03:39:18.332131 20912 raft_consensus.cc:686] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:18.333264 20912 consensus_queue.cc:237] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.341302 20678 raft_consensus.cc:482] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:18.342061 20678 raft_consensus.cc:504] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.343822 20653 raft_consensus.cc:2726] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 03:39:18.355957 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:18.356781 18293 raft_consensus.cc:3037] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:18.355859 20658 leader_election.cc:288] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:18.376781 18140 leader_election.cc:398] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 election: Vote denied by peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163) with higher term. Message: Invalid argument: T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 8. Current term is 9.
I20260807 03:39:18.382223 20658 consensus_queue.cc:1035] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 03:39:18.383250 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:18.350198 21115 raft_consensus.cc:504] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.385710 21115 leader_election.cc:288] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:18.379835 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:18.396256 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:18.396914 18298 raft_consensus.cc:3037] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:18.399894 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:18.400777 18187 raft_consensus.cc:2396] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260807 03:39:18.401528 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:18.407997 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:18.408437 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:18.409037 18190 raft_consensus.cc:2454] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 4.
I20260807 03:39:18.410569 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:18.411731 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:18.412708 18190 raft_consensus.cc:1226] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260807 03:39:18.415850 18309 raft_consensus.cc:1261] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260807 03:39:18.419692 21043 consensus_queue.cc:1035] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
I20260807 03:39:18.425130 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:18.398882 20678 leader_election.cc:288] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:18.430433 18190 raft_consensus.cc:1226] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:18.430698 18189 raft_consensus.cc:1226] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:18.431844 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:18.433166 18293 raft_consensus.cc:2454] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 9.
I20260807 03:39:18.365387 21061 raft_consensus.cc:504] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.434410 18141 leader_election.cc:302] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:18.451517 20653 raft_consensus.cc:2781] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader election won for term 9
I20260807 03:39:18.452576 20653 raft_consensus.cc:686] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:18.453850 20653 consensus_queue.cc:237] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.458060 21061 leader_election.cc:288] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:18.441730 18184 raft_consensus.cc:2454] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3.
I20260807 03:39:18.442205 18183 raft_consensus.cc:2454] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 4.
I20260807 03:39:18.443607 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:18.469962 18190 raft_consensus.cc:2379] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 5: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:18.445109 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:18.445799 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:18.447141 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:18.448267 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:18.449361 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:18.449596 20917 raft_consensus.cc:504] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.438207 18298 raft_consensus.cc:2454] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 7.
I20260807 03:39:18.463219 21043 raft_consensus.cc:482] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:18.465102 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:18.473992 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:18.475015 18141 leader_election.cc:302] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:18.475104 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:18.482360 18295 raft_consensus.cc:2379] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 7: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
W20260807 03:39:18.475688 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 207862us
I20260807 03:39:18.483647 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:18.482081 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:18.484290 18189 raft_consensus.cc:2454] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3.
I20260807 03:39:18.490523 18184 raft_consensus.cc:2354] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 7. Current term is 8.
I20260807 03:39:18.497201 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:18.499352 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:18.503873 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:18.505316 20917 leader_election.cc:288] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:18.506058 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:18.506678 18184 raft_consensus.cc:2454] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 6.
I20260807 03:39:18.507524 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:18.504347 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:18.508389 18190 raft_consensus.cc:2454] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 5.
I20260807 03:39:18.509392 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:18.504966 18183 raft_consensus.cc:1261] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 03:39:18.508808 18248 leader_election.cc:302] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:18.482713 21043 raft_consensus.cc:504] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.511046 20277 raft_consensus.cc:2781] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 03:39:18.511514 20277 raft_consensus.cc:686] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 6 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:18.511932 18248 leader_election.cc:302] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:18.512426 21043 leader_election.cc:288] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:18.513231 20822 raft_consensus.cc:2781] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 03:39:18.513772 20822 raft_consensus.cc:686] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:18.514719 20822 consensus_queue.cc:237] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.516338 18188 raft_consensus.cc:1226] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:18.516769 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:18.517058 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:18.519011 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:18.519078 20277 consensus_queue.cc:237] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.517062 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:18.518277 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:18.520638 18079 raft_consensus.cc:1226] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:18.520700 18298 raft_consensus.cc:3037] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:18.521775 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:18.523108 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:18.524333 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:18.525175 18073 raft_consensus.cc:2379] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 6: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:18.525327 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:18.526018 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ca0223f47b46fd955704eba5259370" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:18.526615 21164 consensus_queue.cc:1046] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:39:18.527266 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:18.527624 21164 raft_consensus.cc:3032] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:39:18.527618 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:18.528081 18180 raft_consensus.cc:2362] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 4. Re-sending same reply.
I20260807 03:39:18.528965 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:18.529850 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:18.530723 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:18.517328 18187 raft_consensus.cc:2362] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3. Re-sending same reply.
I20260807 03:39:18.532017 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:18.532277 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:18.527292 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:18.532889 18190 raft_consensus.cc:2454] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 8.
I20260807 03:39:18.534014 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:18.526031 18193 raft_consensus.cc:2379] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 2: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:18.535645 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:18.528215 21164 raft_consensus.cc:729] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:18.536158 20583 raft_consensus.cc:482] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:18.538548 20583 raft_consensus.cc:504] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.538786 21164 consensus_queue.cc:260] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.539535 18185 raft_consensus.cc:1261] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 03:39:18.540205 21164 raft_consensus.cc:3037] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:18.540680 18185 raft_consensus.cc:3032] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 8 LEADER]: Stepping down as leader of term 8
I20260807 03:39:18.540721 20583 leader_election.cc:288] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:18.541195 18185 raft_consensus.cc:729] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:18.542359 18248 leader_election.cc:302] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:18.491897 20653 raft_consensus.cc:2781] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 03:39:18.536624 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:18.551149 18191 raft_consensus.cc:2454] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 6.
I20260807 03:39:18.551359 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:18.551566 18187 raft_consensus.cc:2454] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 4.
I20260807 03:39:18.552656 18187 raft_consensus.cc:1226] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:18.553021 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:18.554155 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:18.554812 18071 raft_consensus.cc:2354] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 6. Current term is 7.
I20260807 03:39:18.555924 18193 raft_consensus.cc:3037] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 7
I20260807 03:39:18.537812 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:18.531857 18298 raft_consensus.cc:2454] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 8.
I20260807 03:39:18.536417 18027 leader_election.cc:302] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:18.556905 20653 raft_consensus.cc:686] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:18.544042 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:18.517277 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:18.557885 20653 consensus_queue.cc:237] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.548633 20816 raft_consensus.cc:2781] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 03:39:18.559072 20816 raft_consensus.cc:686] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:18.559993 20816 consensus_queue.cc:237] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.560493 21043 raft_consensus.cc:2726] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:18.542035 18185 consensus_queue.cc:260] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.549173 21015 raft_consensus.cc:482] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:18.549379 18082 raft_consensus.cc:1261] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 03:39:18.520655 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:18.563884 21015 raft_consensus.cc:504] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.550072 18072 raft_consensus.cc:1226] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260807 03:39:18.566154 21015 leader_election.cc:288] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:18.566740 18184 raft_consensus.cc:2454] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3.
I20260807 03:39:18.568118 18141 leader_election.cc:302] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:18.569229 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:18.569885 18295 raft_consensus.cc:2454] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 6.
I20260807 03:39:18.571014 18141 leader_election.cc:302] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:18.550866 18080 raft_consensus.cc:1226] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260807 03:39:18.571791 21015 raft_consensus.cc:2781] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader election won for term 8
I20260807 03:39:18.572376 21015 raft_consensus.cc:686] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 8 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:18.571982 18185 raft_consensus.cc:3037] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:18.555678 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:18.536774 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:18.576849 18190 raft_consensus.cc:3037] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 5
I20260807 03:39:18.555351 21166 consensus_queue.cc:1046] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 03:39:18.579378 21035 raft_consensus.cc:3032] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:39:18.573290 21015 consensus_queue.cc:237] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.580626 18183 raft_consensus.cc:1261] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260807 03:39:18.579761 21035 raft_consensus.cc:729] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:18.582152 21035 consensus_queue.cc:260] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.583369 21035 raft_consensus.cc:3037] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:18.584226 20912 raft_consensus.cc:2781] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:39:18.584661 20912 raft_consensus.cc:482] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:18.585047 20912 raft_consensus.cc:3037] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:18.586541 21166 consensus_queue.cc:1035] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 03:39:18.593321 18187 raft_consensus.cc:1261] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260807 03:39:18.594615 21060 consensus_queue.cc:1035] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:39:18.599253 18193 raft_consensus.cc:1261] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260807 03:39:18.579682 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:18.607589 18027 leader_election.cc:302] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:18.610033 18027 leader_election.cc:302] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:18.610829 20583 raft_consensus.cc:2726] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 03:39:18.611490 20583 raft_consensus.cc:2726] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 03:39:18.612493 18184 raft_consensus.cc:1226] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 1. Current term is 2. Ops: []
I20260807 03:39:18.613368 18080 raft_consensus.cc:1226] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 9 LEADER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 8. Current term is 9. Ops: []
I20260807 03:39:18.613808 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:18.615746 18193 raft_consensus.cc:3037] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:18.615311 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:18.611423 18027 leader_election.cc:302] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:18.617785 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:18.618283 21043 raft_consensus.cc:2726] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 03:39:18.618783 18071 raft_consensus.cc:1226] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 6. Current term is 7. Ops: []
I20260807 03:39:18.619928 18069 raft_consensus.cc:1261] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260807 03:39:18.622674 18193 raft_consensus.cc:1261] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 03:39:18.624697 21035 consensus_queue.cc:1035] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:39:18.624851 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:18.631913 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:18.632799 18069 raft_consensus.cc:2454] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 7.
I20260807 03:39:18.634382 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:18.635002 18072 raft_consensus.cc:2454] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 5.
I20260807 03:39:18.636183 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:18.637194 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:18.638136 18082 raft_consensus.cc:2379] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 7: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:18.642565 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:18.626986 17915 catalog_manager.cc:5495] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 3 to 5, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 5 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:18.648362 18185 raft_consensus.cc:1261] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 5. (term mismatch)
I20260807 03:39:18.648854 18185 pending_rounds.cc:76] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02: Aborting all ops after (but not including) 4
I20260807 03:39:18.649295 18185 pending_rounds.cc:97] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20260807 03:39:18.648905 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:18.648221 18072 raft_consensus.cc:2454] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 6.
I20260807 03:39:18.649900 18185 raft_consensus.cc:2866] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 03:39:18.652952 18027 leader_election.cc:302] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:18.654345 18027 leader_election.cc:302] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:18.633706 21060 consensus_queue.cc:1046] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20260807 03:39:18.655354 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:18.655937 18080 raft_consensus.cc:3037] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Advancing to term 9
I20260807 03:39:18.656206 18190 raft_consensus.cc:2454] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 5.
I20260807 03:39:18.662202 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:18.663045 18072 raft_consensus.cc:2454] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 8.
I20260807 03:39:18.644249 17915 catalog_manager.cc:5495] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:39:18.664515 20798 raft_consensus.cc:2726] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:18.670514 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:18.667341 17915 catalog_manager.cc:5495] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:39:18.671321 18082 raft_consensus.cc:2379] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 7: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:18.672175 17915 catalog_manager.cc:5495] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 5 to 8, leader changed from d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 8 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:18.654572 20583 raft_consensus.cc:2726] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260807 03:39:18.674052 17915 catalog_manager.cc:5495] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:18.658254 20822 raft_consensus.cc:3032] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Stepping down as leader of term 1
W20260807 03:39:18.658042 18003 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
I20260807 03:39:18.683120 17915 catalog_manager.cc:5495] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 2 to 5, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 5 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:18.686712 17915 catalog_manager.cc:5495] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 1 to 3, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 3 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:18.691489 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:18.692140 18082 raft_consensus.cc:3037] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Advancing to term 8
I20260807 03:39:18.693456 17915 catalog_manager.cc:5495] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:18.684600 20822 raft_consensus.cc:729] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:18.695242 20822 consensus_queue.cc:260] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.695135 17915 catalog_manager.cc:5495] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 5 to 6, leader changed from d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 6 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:39:18.696521 20822 raft_consensus.cc:3037] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:18.696636 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:18.697490 18072 raft_consensus.cc:2379] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 6: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:18.697695 17915 catalog_manager.cc:5495] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 1 to 3, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 3 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:39:18.695989 18027 leader_election.cc:302] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:18.701370 17915 catalog_manager.cc:5495] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 1 to 2, leader changed from d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 2 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:18.717224 18027 leader_election.cc:302] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:18.703056 18140 leader_election.cc:302] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:18.719449 18082 raft_consensus.cc:2454] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 8.
I20260807 03:39:18.723436 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:18.724200 18082 raft_consensus.cc:2379] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 7: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:18.724540 20583 raft_consensus.cc:2741] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260807 03:39:18.727218 18027 leader_election.cc:302] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:18.719575 21043 raft_consensus.cc:2726] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 03:39:18.727321 17915 catalog_manager.cc:5495] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 5 to 6, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 6 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:39:18.728659 18140 leader_election.cc:302] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:18.729678 18080 raft_consensus.cc:2454] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 9.
W20260807 03:39:18.730850 18003 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 96140f11b8e74293b333cd7e760c3959, TS d9716e2bf06442f5b3b0e81087c53c4d -> TS 5c9b5062bdfa46af9396f1a85025a331 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.683796 20912 raft_consensus.cc:504] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.720498 21015 raft_consensus.cc:2726] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:18.737901 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
W20260807 03:39:18.744189 18003 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 96140f11b8e74293b333cd7e760c3959, TS d9716e2bf06442f5b3b0e81087c53c4d -> TS 5c9b5062bdfa46af9396f1a85025a331 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.738047 18183 raft_consensus.cc:1261] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 03:39:18.738543 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:18.751564 18079 raft_consensus.cc:2454] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 6.
I20260807 03:39:18.747131 18248 leader_election.cc:302] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:18.746996 17915 catalog_manager.cc:5495] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 1 to 2, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 2 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:18.757751 18248 leader_election.cc:302] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:18.744637 20583 raft_consensus.cc:2726] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:18.758955 20822 raft_consensus.cc:3037] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:18.769927 20912 leader_election.cc:288] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:18.770902 18140 leader_election.cc:302] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:18.772152 18079 raft_consensus.cc:1261] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 03:39:18.773947 18070 raft_consensus.cc:1261] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260807 03:39:18.774991 18079 raft_consensus.cc:1261] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260807 03:39:18.776460 21056 consensus_queue.cc:1035] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:39:18.777158 20912 raft_consensus.cc:3037] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:18.762036 20678 raft_consensus.cc:2781] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:39:18.777798 21056 raft_consensus.cc:482] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:18.778393 21056 raft_consensus.cc:504] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.777923 20678 raft_consensus.cc:482] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:18.776660 18079 raft_consensus.cc:1261] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 03:39:18.770104 18075 raft_consensus.cc:1226] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:18.758646 17915 catalog_manager.cc:5495] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 4 to 5, leader changed from d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 5 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:18.772208 18140 leader_election.cc:302] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:18.775856 20822 raft_consensus.cc:2726] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 03:39:18.793421 18070 raft_consensus.cc:1261] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 03:39:18.783062 18081 raft_consensus.cc:2454] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
W20260807 03:39:18.785161 18003 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 91555cf08ec8405c9d002ad0c11f8e5e, TS d9716e2bf06442f5b3b0e81087c53c4d -> TS 5c9b5062bdfa46af9396f1a85025a331 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:18.794613 18003 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 91555cf08ec8405c9d002ad0c11f8e5e, TS d9716e2bf06442f5b3b0e81087c53c4d -> TS 5c9b5062bdfa46af9396f1a85025a331 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.786420 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:18.786557 17915 catalog_manager.cc:5495] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 4 to 6, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 6 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:18.787967 18295 raft_consensus.cc:1261] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 03:39:18.757514 21108 raft_consensus.cc:2726] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:18.796700 17915 catalog_manager.cc:5495] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:18.801824 18081 raft_consensus.cc:1261] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 03:39:18.799396 17915 catalog_manager.cc:5495] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 3 to 5, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 5 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:18.803813 17915 catalog_manager.cc:5495] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:18.782603 20678 raft_consensus.cc:3037] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Advancing to term 7
W20260807 03:39:18.807705 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 50505.2us
I20260807 03:39:18.803222 20658 raft_consensus.cc:2726] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260807 03:39:18.832695 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 60581.9us
I20260807 03:39:18.834565 18027 leader_election.cc:398] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Vote denied by peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057) with higher term. Message: Invalid argument: T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 7. Current term is 8.
I20260807 03:39:18.834940 18027 leader_election.cc:401] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260807 03:39:18.823275 20822 raft_consensus.cc:482] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 03:39:18.837466 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 50380.7us
I20260807 03:39:18.839972 20822 raft_consensus.cc:504] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:18.840785 21182 log.cc:924] Time spent T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.168s user 0.000s sys 0.004s
I20260807 03:39:18.844003 18140 leader_election.cc:302] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:18.809065 20917 consensus_queue.cc:1035] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.027s
I20260807 03:39:18.843508 20798 raft_consensus.cc:2726] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057) with higher term. Message: Invalid argument: T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 7. Current term is 8.
I20260807 03:39:18.849237 21164 consensus_queue.cc:1035] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.014s
I20260807 03:39:18.868189 20822 leader_election.cc:288] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:18.869995 18027 leader_election.cc:302] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:18.851533 21015 raft_consensus.cc:2726] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:18.875424 18027 leader_election.cc:302] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:39:18.877171 18140 leader_election.cc:302] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:18.901736 20917 raft_consensus.cc:2726] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:18.902215 18140 leader_election.cc:302] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:18.882798 20583 raft_consensus.cc:2726] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:18.877660 21056 leader_election.cc:288] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:18.915833 18140 leader_election.cc:398] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Vote denied by peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163) with higher term. Message: Invalid argument: T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 6. Current term is 7.
I20260807 03:39:18.919955 18140 leader_election.cc:302] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:18.909653 20917 raft_consensus.cc:2726] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 03:39:18.930003 21208 consensus_queue.cc:1035] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 03:39:18.931046 18140 leader_election.cc:302] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:18.931404 21108 consensus_queue.cc:1035] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:39:18.921502 20798 raft_consensus.cc:2781] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:39:18.933527 18140 leader_election.cc:302] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:18.934278 20798 raft_consensus.cc:482] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:18.907903 20678 raft_consensus.cc:504] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.943032 18140 leader_election.cc:302] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:18.935201 21015 raft_consensus.cc:2726] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 03:39:18.934755 20798 raft_consensus.cc:3037] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:18.942538 21108 raft_consensus.cc:2726] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:18.946300 18309 raft_consensus.cc:2354] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 6. Current term is 7.
I20260807 03:39:18.946676 21108 raft_consensus.cc:2726] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 03:39:18.947255 18248 leader_election.cc:302] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:18.945533 21208 consensus_queue.cc:1035] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:39:18.949671 21208 consensus_queue.cc:1046] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:39:18.951407 21208 consensus_queue.cc:1035] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:39:18.941416 21115 raft_consensus.cc:2726] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260807 03:39:18.958230 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 82894.6us
I20260807 03:39:18.936230 20653 consensus_queue.cc:1046] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.021s
I20260807 03:39:18.962589 18027 leader_election.cc:302] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:18.965636 20816 raft_consensus.cc:3032] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 03:39:18.966046 20816 raft_consensus.cc:729] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:18.966236 20583 raft_consensus.cc:2726] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:18.966995 20653 raft_consensus.cc:3032] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 03:39:18.967396 20653 raft_consensus.cc:729] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:18.967110 20816 consensus_queue.cc:260] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.968505 20816 raft_consensus.cc:3037] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:18.968221 20653 consensus_queue.cc:260] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:18.969664 20653 raft_consensus.cc:3037] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:18.969856 20583 consensus_queue.cc:1035] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:39:18.970476 20583 consensus_queue.cc:1225] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer 3c4bceb836db4536a0f255fdf86abc02 log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260807 03:39:18.972340 18027 leader_election.cc:302] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:18.966315 20678 leader_election.cc:288] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:18.959865 21166 raft_consensus.cc:2781] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 03:39:18.974411 18141 leader_election.cc:398] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Vote denied by peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021) with higher term. Message: Invalid argument: T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 6. Current term is 7.
I20260807 03:39:18.986354 18141 leader_election.cc:401] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260807 03:39:18.974830 18027 leader_election.cc:302] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:18.977527 20917 consensus_queue.cc:1046] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 03:39:18.977936 18140 leader_election.cc:302] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:18.978219 20658 consensus_queue.cc:1035] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:39:18.986768 20912 raft_consensus.cc:2726] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:18.990695 18027 leader_election.cc:302] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
W20260807 03:39:18.984804 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 136347us
I20260807 03:39:18.988160 21015 raft_consensus.cc:3037] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:18.991515 21043 raft_consensus.cc:2726] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 03:39:18.988168 21166 raft_consensus.cc:686] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
W20260807 03:39:18.988531 21199 log.cc:924] Time spent T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.065s user 0.000s sys 0.001s
I20260807 03:39:19.001617 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:19.003067 18140 leader_election.cc:302] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:18.993150 18027 leader_election.cc:302] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:18.993975 18183 raft_consensus.cc:1226] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:18.996340 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:19.004689 18188 raft_consensus.cc:2454] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 6.
I20260807 03:39:19.000617 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:19.006517 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:19.008147 18140 leader_election.cc:302] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:19.009577 21108 raft_consensus.cc:2726] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 03:39:19.010922 21166 consensus_queue.cc:237] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:19.009589 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:19.012887 18081 raft_consensus.cc:2454] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 6.
I20260807 03:39:18.993683 21214 raft_consensus.cc:2726] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 03:39:19.001395 20912 raft_consensus.cc:2726] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 03:39:19.022257 21108 raft_consensus.cc:2726] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 03:39:19.024952 18190 raft_consensus.cc:1261] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260807 03:39:19.026849 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:19.027426 18073 raft_consensus.cc:3037] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:19.027733 18140 leader_election.cc:302] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:19.030053 20912 raft_consensus.cc:2781] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 03:39:19.030503 20912 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:19.030921 20912 raft_consensus.cc:3037] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Advancing to term 9
W20260807 03:39:19.032876 18003 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 4bd704177fca4d3bbd1b9d39a5f3ec01, TS 66a36e1ac8e945e893668e282d394c49 -> TS 5c9b5062bdfa46af9396f1a85025a331 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:19.033200 18003 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 4bd704177fca4d3bbd1b9d39a5f3ec01, TS 66a36e1ac8e945e893668e282d394c49 -> TS 5c9b5062bdfa46af9396f1a85025a331 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:19.038017 18073 raft_consensus.cc:2454] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 7.
W20260807 03:39:19.042209 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 49159.3us
I20260807 03:39:19.043818 18027 leader_election.cc:302] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:19.048312 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:39:19.045008 21043 raft_consensus.cc:2726] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 03:39:19.055301 20798 raft_consensus.cc:504] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:19.005908 20816 consensus_queue.cc:1035] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
I20260807 03:39:19.059327 21207 raft_consensus.cc:482] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:19.061426 21207 raft_consensus.cc:504] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:19.066062 21108 consensus_queue.cc:1035] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:39:19.069802 18309 raft_consensus.cc:1261] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260807 03:39:19.070248 18309 pending_rounds.cc:76] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49: Aborting all ops after (but not including) 3
I20260807 03:39:19.070506 18309 pending_rounds.cc:97] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260807 03:39:19.070755 18309 raft_consensus.cc:2866] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 03:39:19.078357 18248 leader_election.cc:302] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:19.080138 18071 raft_consensus.cc:1261] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 03:39:19.080709 18080 raft_consensus.cc:1261] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260807 03:39:19.082329 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:19.085008 21189 raft_consensus.cc:482] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:19.085326 18027 leader_election.cc:302] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:19.085487 21189 raft_consensus.cc:504] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:19.086429 18027 leader_election.cc:302] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:19.088356 21213 raft_consensus.cc:2726] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:19.068388 21224 raft_consensus.cc:2781] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader election won for term 4
I20260807 03:39:19.088990 21213 raft_consensus.cc:2726] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:19.089098 21224 raft_consensus.cc:686] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:19.089753 21189 leader_election.cc:288] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:19.090693 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:19.090381 21224 consensus_queue.cc:237] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:19.091274 18072 raft_consensus.cc:2454] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 6.
I20260807 03:39:19.069032 20583 raft_consensus.cc:2726] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:19.093325 20917 consensus_queue.cc:1035] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 03:39:19.097352 21207 leader_election.cc:288] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:19.098196 18295 raft_consensus.cc:1261] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260807 03:39:19.104916 20798 leader_election.cc:288] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:19.105846 19292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331"
I20260807 03:39:19.087926 18027 leader_election.cc:302] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:19.114017 18027 leader_election.cc:302] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:19.116598 18027 leader_election.cc:302] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
W20260807 03:39:19.125763 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 81870.4us
I20260807 03:39:19.127521 18073 raft_consensus.cc:1261] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260807 03:39:19.126330 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:19.131318 18072 raft_consensus.cc:2354] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 6. Current term is 7.
I20260807 03:39:19.131352 18027 leader_election.cc:302] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:19.138166 18069 raft_consensus.cc:1261] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 03:39:19.124634 21214 raft_consensus.cc:2726] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:19.136973 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:19.129717 19292 raft_consensus.cc:2379] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 6: Already voted for candidate 5c9b5062bdfa46af9396f1a85025a331 in this term.
I20260807 03:39:19.146818 18183 raft_consensus.cc:2454] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 6.
I20260807 03:39:19.128041 21061 raft_consensus.cc:2726] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260807 03:39:19.145965 18249 leader_election.cc:302] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
W20260807 03:39:19.152165 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 73889us
I20260807 03:39:19.151057 21043 raft_consensus.cc:2781] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader election won for term 6
I20260807 03:39:19.153309 20658 raft_consensus.cc:3032] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Stepping down as leader of term 6
I20260807 03:39:19.163414 18027 leader_election.cc:302] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:19.154457 21166 raft_consensus.cc:2781] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:39:19.127175 21213 consensus_queue.cc:1035] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20260807 03:39:19.167431 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:39:19.168298 18003 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet b58a8dc36ec94e0c8877bdf0f901598a, TS d9716e2bf06442f5b3b0e81087c53c4d -> TS 5c9b5062bdfa46af9396f1a85025a331 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:19.168659 18003 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet b58a8dc36ec94e0c8877bdf0f901598a, TS d9716e2bf06442f5b3b0e81087c53c4d -> TS 5c9b5062bdfa46af9396f1a85025a331 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:19.168124 18184 raft_consensus.cc:2454] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
I20260807 03:39:19.151355 20917 consensus_queue.cc:1035] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20260807 03:39:19.158784 20816 raft_consensus.cc:2726] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 03:39:19.170295 18249 leader_election.cc:302] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:19.165154 19240 leader_election.cc:302] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:19.174044 21211 consensus_queue.cc:1035] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20260807 03:39:19.166016 21166 raft_consensus.cc:482] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:19.174923 21166 raft_consensus.cc:3037] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:19.175068 21189 raft_consensus.cc:2781] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:39:19.175392 21211 raft_consensus.cc:2781] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:19.175462 21189 raft_consensus.cc:482] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:19.175868 21211 raft_consensus.cc:482] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:19.175954 21189 raft_consensus.cc:3037] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:19.176309 21211 raft_consensus.cc:3037] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:19.180413 21015 raft_consensus.cc:2726] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021) with higher term. Message: Invalid argument: T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 6. Current term is 7.
I20260807 03:39:19.181246 18027 leader_election.cc:302] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:19.162395 20658 raft_consensus.cc:729] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:19.195380 20658 consensus_queue.cc:260] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:19.171770 20798 raft_consensus.cc:2781] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 03:39:19.196496 20658 raft_consensus.cc:3037] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:19.197063 18185 raft_consensus.cc:1261] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 03:39:19.199721 20912 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:19.201351 18027 leader_election.cc:302] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:19.201833 20912 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:19.221275 21061 raft_consensus.cc:2726] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 03:39:19.220711 18188 raft_consensus.cc:2454] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 7.
I20260807 03:39:19.222630 21061 consensus_queue.cc:1035] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:39:19.221477 21213 raft_consensus.cc:2726] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:19.221247 21228 raft_consensus.cc:2741] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
W20260807 03:39:19.252786 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 98683.2us
I20260807 03:39:19.254118 18248 leader_election.cc:302] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:19.257586 18027 leader_election.cc:302] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:19.257488 21242 consensus_queue.cc:1035] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 03:39:19.188964 21015 consensus_queue.cc:1035] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.067s
I20260807 03:39:19.261981 21213 raft_consensus.cc:2726] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 03:39:19.288079 21215 raft_consensus.cc:482] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:19.288492 18309 raft_consensus.cc:1261] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:39:19.288640 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:19.294072 18298 raft_consensus.cc:2454] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 6.
I20260807 03:39:19.292897 18297 raft_consensus.cc:1261] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260807 03:39:19.229838 21166 raft_consensus.cc:504] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:19.282737 21222 raft_consensus.cc:2781] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 03:39:19.298143 21056 consensus_queue.cc:1035] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20260807 03:39:19.303011 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:19.303655 18293 raft_consensus.cc:2379] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 6: Already voted for candidate 5c9b5062bdfa46af9396f1a85025a331 in this term.
I20260807 03:39:19.305500 21189 raft_consensus.cc:504] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:19.313241 21164 consensus_queue.cc:1035] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.002s
I20260807 03:39:19.295570 21043 raft_consensus.cc:686] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 6 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:19.298628 21215 raft_consensus.cc:504] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:19.328953 21214 raft_consensus.cc:482] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:19.329404 21214 raft_consensus.cc:504] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:19.323275 21222 raft_consensus.cc:686] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:19.332063 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:19.332943 18027 leader_election.cc:302] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:19.335386 18069 raft_consensus.cc:3037] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:19.334620 18070 raft_consensus.cc:1261] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 03:39:19.345714 20798 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:19.349654 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:19.350543 20798 raft_consensus.cc:3037] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:19.333091 21222 consensus_queue.cc:237] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:19.358956 21247 raft_consensus.cc:482] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:19.349864 21251 raft_consensus.cc:2781] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:39:19.359622 21247 raft_consensus.cc:504] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:19.360293 21251 raft_consensus.cc:482] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:19.360755 21251 raft_consensus.cc:3037] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:19.361420 21189 leader_election.cc:288] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 7 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:19.362076 21247 leader_election.cc:288] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:19.362710 21166 leader_election.cc:288] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:19.362107 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:19.364560 18183 raft_consensus.cc:3037] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Advancing to term 7
W20260807 03:39:19.365180 21249 log.cc:924] Time spent T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.052s user 0.000s sys 0.001s
I20260807 03:39:19.360777 21043 consensus_queue.cc:237] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:19.369040 21211 raft_consensus.cc:504] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:19.373328 18298 raft_consensus.cc:2454] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 6.
I20260807 03:39:19.378798 18183 raft_consensus.cc:2454] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 7.
I20260807 03:39:19.400509 18069 raft_consensus.cc:2454] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 7.
I20260807 03:39:19.364051 21215 leader_election.cc:288] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:19.377671 21211 leader_election.cc:288] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:19.366751 21214 leader_election.cc:288] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:19.430624 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 248132us
I20260807 03:39:19.392406 21242 consensus_queue.cc:1035] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
I20260807 03:39:19.445087 18027 leader_election.cc:302] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:19.443253 19239 leader_election.cc:302] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:19.462574 21265 consensus_queue.cc:1035] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20260807 03:39:19.463869 21063 raft_consensus.cc:2781] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 03:39:19.465034 21247 raft_consensus.cc:2726] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260807 03:39:19.465126 21257 log.cc:924] Time spent T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20260807 03:39:19.467789 18183 raft_consensus.cc:1261] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 03:39:19.468776 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:19.469980 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:19.470541 18180 raft_consensus.cc:3037] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:19.471843 18188 raft_consensus.cc:1261] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 03:39:19.469430 18184 raft_consensus.cc:2454] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:39:19.472571 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:19.473408 18190 raft_consensus.cc:3037] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:19.477285 18309 raft_consensus.cc:1226] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 6. Current term is 7. Ops: []
I20260807 03:39:19.477735 18295 raft_consensus.cc:1261] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260807 03:39:19.479050 18298 raft_consensus.cc:1261] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260807 03:39:19.461500 21248 consensus_queue.cc:1035] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
W20260807 03:39:19.460379 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 58734.6us
I20260807 03:39:19.500411 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:19.509671 18190 raft_consensus.cc:2454] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 7.
I20260807 03:39:19.502918 18180 raft_consensus.cc:2454] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:19.481972 21224 raft_consensus.cc:482] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:19.503803 20822 consensus_queue.cc:1035] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
I20260807 03:39:19.504971 20658 consensus_queue.cc:1035] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20260807 03:39:19.525017 21240 consensus_queue.cc:1035] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.014s
I20260807 03:39:19.493461 21061 raft_consensus.cc:482] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:19.533519 21061 raft_consensus.cc:504] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:19.529528 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:19.541882 18069 raft_consensus.cc:2454] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
I20260807 03:39:19.529911 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:19.546196 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:19.546984 18298 raft_consensus.cc:2454] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3.
I20260807 03:39:19.548390 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:19.550189 18027 leader_election.cc:302] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:19.554116 18309 raft_consensus.cc:3037] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:19.551846 18248 leader_election.cc:302] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:19.558720 21211 consensus_queue.cc:1035] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:39:19.559026 21060 consensus_queue.cc:1035] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 03:39:19.560288 21060 raft_consensus.cc:2781] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:19.560614 21060 raft_consensus.cc:482] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:19.560925 21060 raft_consensus.cc:3037] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:19.566641 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:19.567131 18082 raft_consensus.cc:3037] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:19.535586 20798 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:19.587343 18309 raft_consensus.cc:2454] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 7.
I20260807 03:39:19.594749 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:19.601174 21255 raft_consensus.cc:2781] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:39:19.601724 21255 raft_consensus.cc:482] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:19.602140 21255 raft_consensus.cc:3037] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:19.603055 18082 raft_consensus.cc:2454] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 7.
I20260807 03:39:19.603291 20798 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:19.604167 21281 log.cc:924] Time spent T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.051s user 0.000s sys 0.004s
I20260807 03:39:19.606592 21251 raft_consensus.cc:504] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:19.514252 21224 raft_consensus.cc:504] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:19.611397 21224 leader_election.cc:288] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:19.614673 21251 leader_election.cc:288] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:19.626709 18248 leader_election.cc:302] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:19.628334 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:19.629033 18309 raft_consensus.cc:2454] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 4.
I20260807 03:39:19.629587 21211 raft_consensus.cc:482] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:19.629050 18079 raft_consensus.cc:1226] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 8. Current term is 9. Ops: []
I20260807 03:39:19.630203 21211 raft_consensus.cc:504] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:19.633105 21260 raft_consensus.cc:2781] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 03:39:19.633744 21260 raft_consensus.cc:686] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:19.634765 21260 consensus_queue.cc:237] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:19.636683 21164 consensus_queue.cc:1046] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260807 03:39:19.637602 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:19.637923 21222 raft_consensus.cc:3032] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 8 LEADER]: Stepping down as leader of term 8
I20260807 03:39:19.638304 21222 raft_consensus.cc:729] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:19.639111 21222 consensus_queue.cc:260] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:19.642964 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 477959us
I20260807 03:39:19.642841 21060 raft_consensus.cc:504] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:19.643926 18140 leader_election.cc:398] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Vote denied by peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163) with higher term. Message: Invalid argument: T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 6. Current term is 7.
I20260807 03:39:19.647800 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:19.653221 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:19.655642 18069 raft_consensus.cc:1226] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 6. Current term is 7. Ops: []
I20260807 03:39:19.656685 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:19.657081 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:19.657439 18180 raft_consensus.cc:2379] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 5: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:19.658308 18072 raft_consensus.cc:1261] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260807 03:39:19.658628 18184 raft_consensus.cc:1261] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260807 03:39:19.660121 18082 raft_consensus.cc:1261] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260807 03:39:19.661284 18188 raft_consensus.cc:1203] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[2.2-5.4] Dedup: 2.2->[4.3-5.4]
I20260807 03:39:19.661718 18188 pending_rounds.cc:76] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02: Aborting all ops after (but not including) 2
I20260807 03:39:19.661932 18188 pending_rounds.cc:97] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260807 03:39:19.662153 18188 raft_consensus.cc:2866] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 03:39:19.666106 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:19.673467 18069 raft_consensus.cc:2454] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
I20260807 03:39:19.674675 18248 leader_election.cc:302] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:19.682044 18027 leader_election.cc:302] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:19.682308 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:19.682927 18183 raft_consensus.cc:2354] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 6. Current term is 7.
I20260807 03:39:19.683698 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:19.684608 18191 raft_consensus.cc:2379] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 3: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:19.688711 18185 raft_consensus.cc:1261] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 03:39:19.654744 21228 raft_consensus.cc:482] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:19.695981 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:19.696833 18190 raft_consensus.cc:2379] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 4: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:19.698047 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:19.713083 21228 raft_consensus.cc:504] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:19.713732 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:19.714839 21211 leader_election.cc:288] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:19.696197 21043 raft_consensus.cc:2781] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:19.654944 21015 consensus_queue.cc:1035] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
I20260807 03:39:19.714331 20653 consensus_queue.cc:1046] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 03:39:19.697517 18140 leader_election.cc:302] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:19.719806 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:19.691025 21035 raft_consensus.cc:482] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:19.687513 20678 raft_consensus.cc:2781] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:19.714255 18311 raft_consensus.cc:1226] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 4. Current term is 5. Ops: []
I20260807 03:39:19.720734 20277 raft_consensus.cc:482] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:19.729810 20912 raft_consensus.cc:482] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:19.746824 21063 raft_consensus.cc:686] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 7 LEADER]: Becoming Leader. State: Replica: 5c9b5062bdfa46af9396f1a85025a331, State: Running, Role: LEADER
W20260807 03:39:19.761314 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 63927.3us
I20260807 03:39:19.758258 20277 raft_consensus.cc:504] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:19.773730 20653 raft_consensus.cc:2726] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260807 03:39:19.745537 21240 raft_consensus.cc:482] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:19.734812 21043 raft_consensus.cc:482] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:19.774878 21108 raft_consensus.cc:3032] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Stepping down as leader of term 6
I20260807 03:39:19.775197 21060 leader_election.cc:288] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:19.775358 21108 raft_consensus.cc:729] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:19.746381 21255 raft_consensus.cc:504] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:19.776599 21108 consensus_queue.cc:260] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:19.777768 21255 leader_election.cc:288] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:19.774672 20277 leader_election.cc:288] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:19.774478 21273 consensus_queue.cc:1035] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 03:39:19.778007 21108 raft_consensus.cc:3037] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:19.747313 20678 raft_consensus.cc:482] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:19.789131 21224 consensus_queue.cc:1046] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260807 03:39:19.790086 21224 raft_consensus.cc:3032] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 LEADER]: Stepping down as leader of term 4
I20260807 03:39:19.790436 21224 raft_consensus.cc:729] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:19.738648 21265 consensus_queue.cc:1035] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.044s
I20260807 03:39:19.775146 21043 raft_consensus.cc:3037] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:19.777912 20912 raft_consensus.cc:504] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:19.751709 21035 raft_consensus.cc:504] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:19.805969 21061 leader_election.cc:288] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:19.807307 21035 leader_election.cc:288] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:19.813788 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:19.792536 20678 raft_consensus.cc:3037] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:19.817634 18297 raft_consensus.cc:2354] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 3. Current term is 4.
I20260807 03:39:19.790709 21063 consensus_queue.cc:237] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:19.865549 18027 leader_election.cc:398] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Vote denied by peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021) with higher term. Message: Invalid argument: T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 3. Current term is 4.
I20260807 03:39:19.865972 18027 leader_election.cc:401] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260807 03:39:19.774896 21240 raft_consensus.cc:504] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:19.948047 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:19.927410 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 163995us
I20260807 03:39:19.791271 21224 consensus_queue.cc:260] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:19.868479 21247 raft_consensus.cc:3037] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:19.981871 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:20.004760 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:20.001585 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:20.010813 18075 raft_consensus.cc:2454] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:20.013325 18027 leader_election.cc:398] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 election: Vote denied by peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057) with higher term. Message: Invalid argument: T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 6. Current term is 7.
I20260807 03:39:19.985432 21224 raft_consensus.cc:3037] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:20.089552 18027 leader_election.cc:302] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:20.100941 18293 raft_consensus.cc:1261] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260807 03:39:20.134532 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 45071.2us
I20260807 03:39:20.135514 18027 leader_election.cc:302] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:20.099591 21166 raft_consensus.cc:482] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 03:39:20.139616 21270 log.cc:924] Time spent T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
W20260807 03:39:20.143391 21307 log.cc:924] Time spent T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.088s user 0.001s sys 0.000s
W20260807 03:39:20.154809 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 80192us
I20260807 03:39:20.130182 21265 consensus_queue.cc:1035] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.049s
I20260807 03:39:20.162701 18248 leader_election.cc:302] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
W20260807 03:39:20.171104 21311 log.cc:924] Time spent T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
W20260807 03:39:20.173796 21304 log.cc:924] Time spent T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.264s user 0.001s sys 0.000s
I20260807 03:39:20.148972 20912 leader_election.cc:288] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:20.240656 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 71088.9us
I20260807 03:39:20.199633 20277 raft_consensus.cc:2781] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:39:20.246193 20277 raft_consensus.cc:482] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:20.246508 20277 raft_consensus.cc:3037] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Advancing to term 7
W20260807 03:39:20.167538 19240 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 172599us
I20260807 03:39:20.268720 18184 raft_consensus.cc:1226] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 4. Current term is 5. Ops: []
I20260807 03:39:20.274380 18069 raft_consensus.cc:1261] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260807 03:39:20.148798 21166 raft_consensus.cc:504] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:20.279759 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:20.288250 18180 raft_consensus.cc:2379] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 3: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
W20260807 03:39:20.289816 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 735879us
I20260807 03:39:20.298985 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:20.274147 21218 raft_consensus.cc:482] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 03:39:20.286684 21306 log.cc:924] Time spent T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.335s user 0.001s sys 0.000s
I20260807 03:39:20.318037 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:20.318900 18072 raft_consensus.cc:2396] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260807 03:39:20.274070 20912 raft_consensus.cc:482] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:20.278342 21214 raft_consensus.cc:482] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:20.321919 18185 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
mode: GRACEFUL
new_leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:34758
I20260807 03:39:20.325301 18185 raft_consensus.cc:595] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Received request to transfer leadership to TS d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:20.319080 18249 leader_election.cc:302] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:20.343659 18185 raft_consensus.cc:603] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 03:39:20.295698 21240 leader_election.cc:288] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:20.176018 21332 consensus_queue.cc:1035] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.155s
I20260807 03:39:20.298781 21265 raft_consensus.cc:482] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:20.362445 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:20.324223 21251 raft_consensus.cc:482] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:20.388883 20277 raft_consensus.cc:504] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:20.370466 20822 raft_consensus.cc:2781] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 03:39:20.342861 20912 raft_consensus.cc:504] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:20.367920 21341 raft_consensus.cc:482] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:20.347580 21214 raft_consensus.cc:504] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:20.321883 21218 raft_consensus.cc:504] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:20.412981 20277 leader_election.cc:288] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:20.382054 21329 raft_consensus.cc:2726] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260807 03:39:20.418104 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 99140.9us
I20260807 03:39:20.425040 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:20.425844 18072 raft_consensus.cc:3037] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 7
I20260807 03:39:20.375200 21251 raft_consensus.cc:504] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:20.363842 21265 raft_consensus.cc:504] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:20.427130 18293 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8c98c1d422654739af06653a512f7499"
dest_uuid: "66a36e1ac8e945e893668e282d394c49"
mode: GRACEFUL
new_leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:58736
I20260807 03:39:20.440729 18293 raft_consensus.cc:595] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Received request to transfer leadership to TS d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:20.441148 18293 raft_consensus.cc:603] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260807 03:39:20.429651 21314 log.cc:924] Time spent T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.305s user 0.001s sys 0.000s
I20260807 03:39:20.430811 21302 raft_consensus.cc:482] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:20.420710 21336 consensus_queue.cc:1035] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.108s
I20260807 03:39:20.432395 21214 leader_election.cc:288] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:20.432727 21323 log.cc:924] Time spent T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.279s user 0.001s sys 0.000s
W20260807 03:39:20.432791 21330 log.cc:924] Time spent T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.085s user 0.002s sys 0.000s
I20260807 03:39:20.414269 20822 raft_consensus.cc:686] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
W20260807 03:39:20.437093 21322 log.cc:924] Time spent T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20260807 03:39:20.437197 21222 raft_consensus.cc:3037] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:20.427594 21221 raft_consensus.cc:482] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:20.410761 21320 raft_consensus.cc:482] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:20.439766 21166 leader_election.cc:288] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:20.449210 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:20.472752 21277 raft_consensus.cc:482] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:20.473241 18079 raft_consensus.cc:2454] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:39:20.473366 21277 raft_consensus.cc:504] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:20.474515 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:20.475245 18079 raft_consensus.cc:2454] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 1.
I20260807 03:39:20.475647 21277 leader_election.cc:288] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
W20260807 03:39:20.443749 21043 consensus_meta.cc:287] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 0.603s user 0.009s sys 0.002s
I20260807 03:39:20.476414 21043 raft_consensus.cc:504] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:20.478816 21043 leader_election.cc:288] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:20.481122 21320 raft_consensus.cc:504] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:20.481931 20912 leader_election.cc:288] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:20.483467 21265 leader_election.cc:288] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:20.416016 21341 raft_consensus.cc:504] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:20.484297 21221 raft_consensus.cc:504] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:20.485890 21341 leader_election.cc:288] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:20.486954 21221 leader_election.cc:288] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:20.487258 21251 leader_election.cc:288] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:20.489223 20822 consensus_queue.cc:237] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:20.491657 18295 raft_consensus.cc:2454] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 9.
I20260807 03:39:20.493180 18187 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
mode: GRACEFUL
new_leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:34758
I20260807 03:39:20.493752 18187 raft_consensus.cc:595] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Received request to transfer leadership to TS d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:20.494057 18187 raft_consensus.cc:603] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 03:39:20.497067 21352 raft_consensus.cc:482] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:20.497682 21352 raft_consensus.cc:504] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:20.501343 20678 consensus_meta.cc:287] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.642s user 0.010s sys 0.000s
I20260807 03:39:20.501753 20678 raft_consensus.cc:504] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:20.439803 21218 leader_election.cc:288] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
W20260807 03:39:20.519008 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 46741.3us
I20260807 03:39:20.483772 21320 leader_election.cc:288] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:20.460309 21302 raft_consensus.cc:504] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:20.507512 18080 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33254 (request call id 1360) took 1425 ms. Trace:
I20260807 03:39:20.508587 21228 leader_election.cc:288] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:20.510676 21352 leader_election.cc:288] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:20.463864 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 321901us
I20260807 03:39:20.511607 21271 raft_consensus.cc:482] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:20.513504 20912 raft_consensus.cc:482] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:20.517287 18248 leader_election.cc:302] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:20.517882 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:20.473552 21338 log.cc:924] Time spent T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20260807 03:39:20.522419 18080 rpcz_store.cc:276] 0807 03:39:19.081930 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:19.082080 (+ 150us) service_pool.cc:227] Handling call
0807 03:39:20.507308 (+1425228us) spinlock_profiling.cc:243] Waited 349 ms on lock 0x7b54003d2cd4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:20.507433 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":801724800}
I20260807 03:39:20.526839 18079 raft_consensus.cc:2454] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 6.
I20260807 03:39:20.526147 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:20.530004 18248 leader_election.cc:302] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:20.532730 18069 raft_consensus.cc:2379] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 5: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
W20260807 03:39:20.522419 21108 consensus_meta.cc:287] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 0.706s user 0.007s sys 0.004s
I20260807 03:39:20.557945 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:20.541421 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:20.543864 18188 raft_consensus.cc:1261] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 03:39:20.547713 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:20.561604 18185 raft_consensus.cc:2379] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 4: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:20.548615 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:20.548802 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:20.532482 21271 raft_consensus.cc:504] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:20.551245 21247 consensus_meta.cc:287] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 0.589s user 0.006s sys 0.003s
I20260807 03:39:20.552691 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:20.565775 18190 raft_consensus.cc:2379] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 4: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:20.552984 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:20.568653 18180 raft_consensus.cc:2396] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260807 03:39:20.553402 20678 leader_election.cc:288] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:20.553918 21336 raft_consensus.cc:2781] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:39:20.533354 21277 raft_consensus.cc:2781] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:20.554663 18248 leader_election.cc:302] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:20.555150 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:20.556272 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:20.557199 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:20.563171 18081 raft_consensus.cc:2454] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 7.
I20260807 03:39:20.557230 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:20.571097 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:20.554078 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:20.573918 18311 raft_consensus.cc:2454] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 2.
I20260807 03:39:20.574807 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:20.586943 18310 raft_consensus.cc:2396] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260807 03:39:20.587100 18184 raft_consensus.cc:2379] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 2: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:20.596833 18140 leader_election.cc:302] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:20.551725 20912 raft_consensus.cc:504] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:20.709024 21352 raft_consensus.cc:2781] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 03:39:20.737790 21352 raft_consensus.cc:482] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:20.738214 21352 raft_consensus.cc:3037] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:20.739056 20912 leader_election.cc:288] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:20.705295 21277 raft_consensus.cc:482] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:20.740831 21277 raft_consensus.cc:3037] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:20.742256 21341 raft_consensus.cc:482] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:20.742738 21341 raft_consensus.cc:504] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:20.743778 18312 raft_consensus.cc:2454] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 4.
I20260807 03:39:20.744509 21364 raft_consensus.cc:482] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:20.744869 21341 leader_election.cc:288] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:20.745111 21364 raft_consensus.cc:504] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:20.745440 21341 raft_consensus.cc:482] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:20.745433 18072 raft_consensus.cc:2454] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 7.
I20260807 03:39:20.746101 21341 raft_consensus.cc:504] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:20.721848 21302 leader_election.cc:288] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:20.564656 21219 raft_consensus.cc:482] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:20.533584 21265 consensus_queue.cc:1035] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.189s
I20260807 03:39:20.569372 21247 raft_consensus.cc:2726] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021) with higher term. Message: Invalid argument: T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 3. Current term is 4.
I20260807 03:39:20.725023 21271 leader_election.cc:288] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:20.730429 18310 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd"
dest_uuid: "66a36e1ac8e945e893668e282d394c49"
mode: GRACEFUL
new_leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:58736
I20260807 03:39:20.775239 18310 raft_consensus.cc:595] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Received request to transfer leadership to TS d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:20.775699 18310 raft_consensus.cc:603] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 03:39:20.731242 18027 leader_election.cc:302] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:20.737133 18189 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44662 (request call id 872) took 1079 ms. Trace:
I20260807 03:39:20.740171 20678 raft_consensus.cc:2781] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:39:20.747498 21334 raft_consensus.cc:482] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:20.747733 21340 raft_consensus.cc:482] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:20.776901 18189 rpcz_store.cc:276] 0807 03:39:19.657507 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:19.657622 (+ 115us) service_pool.cc:227] Handling call
0807 03:39:19.679890 (+ 22268us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:19.680288 (+ 398us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:19.680328 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:39:19.695813 (+ 15485us) log.cc:842] Serialized 26 byte log entry
0807 03:39:19.890734 (+194921us) raft_consensus.cc:1631] Marking committed up to 4
0807 03:39:19.981683 (+ 90949us) log.cc:842] Serialized 14 byte log entry
0807 03:39:20.001723 (+ 20040us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:20.001792 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:20.734273 (+732481us) spinlock_profiling.cc:243] Waited 65 ms on lock 0x7b18003e7040. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe9981368b1 00007fe998143661 00007fe98f297638 00007fe98f29ab11 00007fe98f29aac9 00007fe98f29aa59 00007fe98f29aa21 00007fe98f299d1d 00007fe9984fb204 00007fe9984fb039 00007fe98f279dc6
0807 03:39:20.736728 (+ 2455us) raft_consensus.cc:1671] finished
0807 03:39:20.736863 (+ 135us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:20.737037 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":95904,"prepare.queue_time_us":91258,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":3421,"raft.queue_time_us":175641,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":4630,"spinlock_wait_cycles":149490944,"thread_start_us":137079,"threads_started":1,"wal-append.queue_time_us":276310}
I20260807 03:39:20.748054 21240 raft_consensus.cc:482] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:20.704408 21336 raft_consensus.cc:482] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:20.747495 21239 raft_consensus.cc:482] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:20.709784 21320 raft_consensus.cc:482] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:20.749675 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:20.710189 20917 raft_consensus.cc:482] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:20.750317 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:20.752166 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:20.740017 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 185413us
W20260807 03:39:20.823837 18141 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.724s user 0.022s sys 0.041s
W20260807 03:39:20.824255 18141 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.734s user 0.022s sys 0.041s
I20260807 03:39:20.824536 18248 leader_election.cc:302] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:20.770359 20912 raft_consensus.cc:482] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
W20260807 03:39:20.771556 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 452493us
I20260807 03:39:20.747382 21364 leader_election.cc:288] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:20.780725 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:20.706511 21108 raft_consensus.cc:482] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:20.818730 18295 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57276 (request call id 945) took 1182 ms. Trace:
W20260807 03:39:20.822013 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 90854.1us
I20260807 03:39:20.827366 18079 raft_consensus.cc:2379] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 4: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:20.804260 20678 raft_consensus.cc:482] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 03:39:20.773352 19240 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 403795us
I20260807 03:39:20.849030 18295 rpcz_store.cc:276] 0807 03:39:19.627372 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:19.647549 (+ 20177us) service_pool.cc:227] Handling call
0807 03:39:20.810007 (+1162458us) spinlock_profiling.cc:243] Waited 836 ms on lock 0x7b54001d0454. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:20.810119 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1922118912}
I20260807 03:39:20.845692 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:20.848563 18069 raft_consensus.cc:2379] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 7: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:20.862515 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:20.898926 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:20.866459 18185 raft_consensus.cc:1261] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260807 03:39:20.902611 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 46530.5us
I20260807 03:39:20.821949 21336 raft_consensus.cc:3037] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:20.818817 21340 raft_consensus.cc:504] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:20.907548 21043 consensus_queue.cc:1035] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 03:39:20.862046 21108 raft_consensus.cc:504] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:20.844928 21115 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.839s user 0.015s sys 0.048s
I20260807 03:39:20.909287 18295 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a"
dest_uuid: "66a36e1ac8e945e893668e282d394c49"
mode: GRACEFUL
new_leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:58736
I20260807 03:39:20.830766 21251 raft_consensus.cc:2726] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:20.926393 18295 raft_consensus.cc:595] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Received request to transfer leadership to TS d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:20.926800 18295 raft_consensus.cc:603] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 03:39:20.915804 18081 raft_consensus.cc:1261] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260807 03:39:20.921084 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:20.922214 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:20.924767 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:20.844724 20798 raft_consensus.cc:482] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:20.861794 21218 raft_consensus.cc:2781] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 03:39:20.934228 18072 raft_consensus.cc:3037] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:20.772879 21219 raft_consensus.cc:504] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:20.822068 21240 raft_consensus.cc:504] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:20.947199 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:20.869450 21364 raft_consensus.cc:482] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:20.939963 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260807 03:39:20.940131 20798 raft_consensus.cc:504] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:20.947729 18312 raft_consensus.cc:2454] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 7.
I20260807 03:39:20.958395 18070 raft_consensus.cc:2454] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
I20260807 03:39:20.830947 21239 raft_consensus.cc:504] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:20.949169 21337 raft_consensus.cc:482] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
W20260807 03:39:20.926229 21115 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.923s user 0.020s sys 0.053s
I20260807 03:39:20.832422 21320 raft_consensus.cc:504] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:20.832271 20917 raft_consensus.cc:504] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:20.866629 20678 raft_consensus.cc:3037] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:20.929344 21265 raft_consensus.cc:482] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:20.962963 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:20.964319 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:20.985363 18312 raft_consensus.cc:2354] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 4. Current term is 5.
I20260807 03:39:20.975369 18075 raft_consensus.cc:1261] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260807 03:39:20.969811 18082 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.605s user 0.027s sys 0.051s
W20260807 03:39:20.986675 18082 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.622s user 0.027s sys 0.051s
I20260807 03:39:20.976459 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:20.848625 20912 raft_consensus.cc:504] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:20.981384 21219 leader_election.cc:288] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:20.981768 21340 leader_election.cc:288] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:20.982906 21240 leader_election.cc:288] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:20.984324 20798 leader_election.cc:288] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:20.991036 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:20.994546 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:21.003993 18190 raft_consensus.cc:2454] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:39:20.820752 21334 raft_consensus.cc:504] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:20.988157 20583 raft_consensus.cc:2726] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 03:39:21.011489 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:20.991488 21218 raft_consensus.cc:686] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:20.961043 21364 raft_consensus.cc:504] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:20.993276 20816 raft_consensus.cc:482] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:21.031164 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:21.031939 18190 raft_consensus.cc:2454] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 6.
I20260807 03:39:20.991706 21265 raft_consensus.cc:504] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:21.016777 21341 leader_election.cc:288] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:21.013003 21221 raft_consensus.cc:482] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:21.061870 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
W20260807 03:39:21.058456 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 52019.6us
I20260807 03:39:21.063441 18190 raft_consensus.cc:3037] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
W20260807 03:39:21.067076 21370 log.cc:924] Time spent T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331: Append to log took a long time: real 0.082s user 0.001s sys 0.000s
I20260807 03:39:20.972680 21337 raft_consensus.cc:504] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:21.070137 21337 leader_election.cc:288] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:21.070228 21320 leader_election.cc:288] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:21.073148 20912 leader_election.cc:288] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:21.029457 20816 raft_consensus.cc:504] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:21.074585 21364 leader_election.cc:288] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:21.075301 21218 consensus_queue.cc:237] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:21.075238 20816 leader_election.cc:288] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:21.080725 18072 raft_consensus.cc:2454] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:21.084584 21108 leader_election.cc:288] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:21.099970 18248 leader_election.cc:302] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:21.118880 18190 raft_consensus.cc:2454] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
I20260807 03:39:21.119894 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:21.092875 21352 raft_consensus.cc:504] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:21.133270 19240 leader_election.cc:302] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:21.134508 21302 raft_consensus.cc:2726] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:21.135167 21239 leader_election.cc:288] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:21.136214 21378 log.cc:924] Time spent T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20260807 03:39:21.136713 21277 raft_consensus.cc:504] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:21.138123 18298 raft_consensus.cc:2354] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 4. Current term is 5.
I20260807 03:39:21.082639 21221 raft_consensus.cc:504] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:21.130906 20822 raft_consensus.cc:2781] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 03:39:21.115823 21166 consensus_queue.cc:1035] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.045s
I20260807 03:39:21.124341 21215 raft_consensus.cc:482] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:21.149650 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:21.154536 18183 raft_consensus.cc:2454] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 6.
I20260807 03:39:21.156198 21265 leader_election.cc:288] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:21.157702 21334 leader_election.cc:288] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:21.158429 20917 leader_election.cc:288] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:21.195132 21336 raft_consensus.cc:504] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:21.121677 21260 consensus_queue.cc:1035] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.029s
I20260807 03:39:21.131925 20678 raft_consensus.cc:504] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:21.167263 20822 raft_consensus.cc:686] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:21.161289 21215 raft_consensus.cc:504] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:21.222175 21215 leader_election.cc:288] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:21.224269 20678 leader_election.cc:288] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:21.234747 21273 raft_consensus.cc:482] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:21.241006 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:21.241295 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:21.242389 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:21.243000 18305 raft_consensus.cc:2454] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:39:21.243165 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ca0223f47b46fd955704eba5259370" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:21.243887 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:21.244239 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:21.244632 18305 raft_consensus.cc:2379] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 7: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:21.244758 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:21.246044 21365 raft_consensus.cc:482] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:21.246568 21365 raft_consensus.cc:504] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:21.247107 20653 raft_consensus.cc:482] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:21.247328 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:21.247759 21271 raft_consensus.cc:482] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:21.247640 20653 raft_consensus.cc:504] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:21.248724 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:21.249596 20653 leader_election.cc:288] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:21.249655 21271 raft_consensus.cc:504] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:21.251798 21271 leader_election.cc:288] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:21.252396 21271 raft_consensus.cc:482] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:21.252910 21271 raft_consensus.cc:504] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:21.254860 21271 leader_election.cc:288] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:21.255910 21366 raft_consensus.cc:482] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:21.256528 21366 raft_consensus.cc:504] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:21.257648 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:21.258723 21366 leader_election.cc:288] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:21.260053 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:21.260901 21336 leader_election.cc:288] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:21.263660 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:21.263818 18297 raft_consensus.cc:1261] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Refusing update from remote peer 5c9b5062bdfa46af9396f1a85025a331: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260807 03:39:21.265290 21221 leader_election.cc:288] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:21.267371 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:21.274554 18193 raft_consensus.cc:2454] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
I20260807 03:39:21.275040 18183 raft_consensus.cc:1226] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 8. Current term is 9. Ops: []
I20260807 03:39:21.275138 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:21.275735 18073 raft_consensus.cc:2454] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 5.
I20260807 03:39:21.275823 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:21.276387 18193 raft_consensus.cc:1261] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260807 03:39:21.267673 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:21.277009 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:21.277408 18187 raft_consensus.cc:2454] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 2.
I20260807 03:39:21.269129 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:21.277720 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:21.269414 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 240072us
I20260807 03:39:21.278154 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:21.278822 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:21.270408 20798 raft_consensus.cc:482] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:21.278185 18190 raft_consensus.cc:2454] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 6.
I20260807 03:39:21.279415 18081 raft_consensus.cc:2454] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:39:21.268838 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:21.279702 20798 raft_consensus.cc:504] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:21.271236 21349 raft_consensus.cc:482] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:21.280242 21352 leader_election.cc:288] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:21.280648 21349 raft_consensus.cc:504] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:21.281498 21277 leader_election.cc:288] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:21.281767 20798 leader_election.cc:288] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:21.282773 21349 leader_election.cc:288] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:21.283007 21312 log.cc:924] Time spent T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 1.139s user 0.014s sys 0.000s
I20260807 03:39:21.270648 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:21.278825 18027 leader_election.cc:302] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:21.284665 18191 raft_consensus.cc:3037] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 5
I20260807 03:39:21.285488 18027 leader_election.cc:302] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:21.279299 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:21.286432 18082 raft_consensus.cc:2454] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 6.
I20260807 03:39:21.286597 18027 leader_election.cc:302] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:21.286998 20583 raft_consensus.cc:482] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:21.280328 18184 raft_consensus.cc:2454] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:21.282680 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ca0223f47b46fd955704eba5259370" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:21.271709 21363 raft_consensus.cc:482] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:21.257922 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:39:21.288259 21224 raft_consensus.cc:2726] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:21.288785 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:39:21.288952 21224 raft_consensus.cc:2726] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:21.289197 21213 raft_consensus.cc:2781] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader election won for term 4
I20260807 03:39:21.289746 21213 raft_consensus.cc:686] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:21.290546 21213 consensus_queue.cc:237] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:21.288342 21363 raft_consensus.cc:504] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:21.293232 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:21.293494 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:21.294158 18180 raft_consensus.cc:3037] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:21.272903 20822 consensus_queue.cc:237] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:21.296592 18070 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33254 (request call id 1386) took 1315 ms. Trace:
I20260807 03:39:21.287515 20583 raft_consensus.cc:504] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:21.301487 18082 raft_consensus.cc:1261] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Refusing update from remote peer 5c9b5062bdfa46af9396f1a85025a331: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260807 03:39:21.277130 18183 raft_consensus.cc:1261] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260807 03:39:21.287758 18027 leader_election.cc:302] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:21.305523 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:21.296921 18070 rpcz_store.cc:276] 0807 03:39:19.981476 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:19.981613 (+ 137us) service_pool.cc:227] Handling call
0807 03:39:21.296449 (+1314836us) spinlock_profiling.cc:243] Waited 793 ms on lock 0x7b5400340b54. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:21.296543 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1824364288}
I20260807 03:39:21.309357 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:21.309986 18070 raft_consensus.cc:3037] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:21.319082 18070 raft_consensus.cc:2454] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 7.
I20260807 03:39:21.283846 21365 leader_election.cc:288] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:21.334489 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:21.334077 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:21.335852 18070 raft_consensus.cc:3037] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 3
I20260807 03:39:21.322558 21224 raft_consensus.cc:2781] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader election won for term 9
I20260807 03:39:21.337306 21224 raft_consensus.cc:686] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 9 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:21.337623 21335 consensus_queue.cc:1035] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 03:39:21.338186 21224 consensus_queue.cc:237] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:21.338652 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:21.339334 18081 raft_consensus.cc:2379] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 7: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:21.293433 21363 leader_election.cc:288] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:21.343559 18027 leader_election.cc:302] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:21.345336 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:21.348978 18027 leader_election.cc:398] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Vote denied by peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021) with higher term. Message: Invalid argument: T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 4. Current term is 5.
I20260807 03:39:21.349280 18027 leader_election.cc:401] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260807 03:39:21.351007 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:21.357239 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:21.357514 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:21.357828 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:21.358179 21063 consensus_queue.cc:1035] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 03:39:21.358590 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:21.358915 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:21.359689 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:21.308429 21239 raft_consensus.cc:482] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:21.368777 18070 raft_consensus.cc:2454] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:39:21.361830 18180 raft_consensus.cc:2454] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 7.
I20260807 03:39:21.360375 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:21.372457 18191 raft_consensus.cc:2454] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:21.373553 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:21.235497 21273 raft_consensus.cc:504] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:21.404778 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:21.364269 20798 raft_consensus.cc:3037] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 5
I20260807 03:39:21.386438 21349 raft_consensus.cc:2741] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260807 03:39:21.426657 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:21.430172 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:21.430929 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:21.436339 18191 raft_consensus.cc:3037] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:21.436558 18180 raft_consensus.cc:2454] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 7.
I20260807 03:39:21.431706 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:21.437618 18183 raft_consensus.cc:3037] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:21.412623 21265 raft_consensus.cc:482] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:21.464372 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:21.466696 18187 raft_consensus.cc:3037] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:21.470142 18191 raft_consensus.cc:2454] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:39:21.414326 17912 catalog_manager.cc:5495] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 reported cstate change: term changed from 5 to 7, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132). New cstate: current_term: 7 leader_uuid: "5c9b5062bdfa46af9396f1a85025a331" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 03:39:21.388594 21239 raft_consensus.cc:504] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:21.496807 18183 raft_consensus.cc:2454] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
W20260807 03:39:21.527530 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 51493.8us
I20260807 03:39:21.496871 18191 raft_consensus.cc:1261] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Refusing update from remote peer 5c9b5062bdfa46af9396f1a85025a331: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260807 03:39:21.528862 18249 leader_election.cc:302] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:21.530122 18249 leader_election.cc:302] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:21.532984 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
W20260807 03:39:21.529318 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 76883us
I20260807 03:39:21.475188 21265 raft_consensus.cc:504] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:21.546206 18141 leader_election.cc:302] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:21.548799 20917 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:21.553449 18249 leader_election.cc:302] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:21.553617 21406 consensus_queue.cc:1035] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 03:39:21.554744 18027 leader_election.cc:302] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:21.555857 21273 leader_election.cc:288] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:21.556696 20583 leader_election.cc:288] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:21.580194 18141 leader_election.cc:302] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:21.582715 18027 leader_election.cc:302] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:21.540561 21336 raft_consensus.cc:2781] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 03:39:21.543591 21223 raft_consensus.cc:2781] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:39:21.588474 18141 leader_election.cc:302] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:21.593320 18141 leader_election.cc:302] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:21.595300 18027 leader_election.cc:302] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:21.597295 18249 leader_election.cc:302] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:21.600445 18141 leader_election.cc:398] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021) with higher term. Message: Invalid argument: T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 4. Current term is 5.
I20260807 03:39:21.600778 18141 leader_election.cc:401] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260807 03:39:21.606629 18249 leader_election.cc:302] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:21.606621 21336 raft_consensus.cc:686] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:21.608170 20816 raft_consensus.cc:2781] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:39:21.608700 20816 raft_consensus.cc:686] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:21.608439 21336 consensus_queue.cc:237] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:21.609575 20816 consensus_queue.cc:237] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:21.606853 21060 raft_consensus.cc:2781] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:39:21.612320 21060 raft_consensus.cc:482] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:21.612715 21060 raft_consensus.cc:3037] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:21.613013 21367 raft_consensus.cc:2726] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:21.607091 21247 raft_consensus.cc:2726] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:21.592942 20822 raft_consensus.cc:2781] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:21.613612 18249 leader_election.cc:302] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:21.584491 21365 raft_consensus.cc:2726] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:21.617620 18141 leader_election.cc:302] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:21.619004 18027 leader_election.cc:302] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:21.618719 20822 raft_consensus.cc:482] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:21.621237 20822 raft_consensus.cc:3037] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:21.594811 21219 raft_consensus.cc:2726] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:21.624656 20277 raft_consensus.cc:2781] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 03:39:21.536976 21391 consensus_queue.cc:1035] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.073s
I20260807 03:39:21.601811 21273 raft_consensus.cc:2726] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:21.595912 21223 raft_consensus.cc:482] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:21.627554 20277 raft_consensus.cc:686] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:21.584013 20917 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:21.626080 18141 leader_election.cc:302] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
W20260807 03:39:21.708328 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 82315.1us
I20260807 03:39:21.709254 18141 leader_election.cc:302] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:21.711272 18141 leader_election.cc:302] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
W20260807 03:39:21.678529 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 59598.5us
W20260807 03:39:21.682168 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 53923.3us
I20260807 03:39:21.604377 21366 raft_consensus.cc:2726] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 03:39:21.682090 20277 consensus_queue.cc:237] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:21.712587 18027 leader_election.cc:302] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:21.716862 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:21.673581 21365 raft_consensus.cc:2726] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:21.719909 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:21.683684 21223 raft_consensus.cc:3037] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:21.730376 18141 leader_election.cc:302] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:21.684921 20583 raft_consensus.cc:2726] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:21.734898 18027 leader_election.cc:302] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:21.614362 21063 consensus_queue.cc:1035] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.106s
I20260807 03:39:21.739218 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:21.739055 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:21.741411 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:21.741634 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:21.742179 18295 raft_consensus.cc:3037] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:21.743556 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:21.717753 21329 raft_consensus.cc:2726] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 03:39:21.682890 21373 raft_consensus.cc:2726] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260807 03:39:21.713833 21271 raft_consensus.cc:2726] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:21.758960 18295 raft_consensus.cc:2454] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 8.
I20260807 03:39:21.760319 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:21.750079 21417 log.cc:924] Time spent T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.087s user 0.000s sys 0.000s
I20260807 03:39:21.764865 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:21.768513 18141 leader_election.cc:302] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:21.727914 21408 raft_consensus.cc:2726] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021) with higher term. Message: Invalid argument: T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 4. Current term is 5.
I20260807 03:39:21.772310 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:21.772605 21384 raft_consensus.cc:2726] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:21.746945 21214 raft_consensus.cc:2726] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:21.747641 21273 raft_consensus.cc:2726] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 03:39:21.786820 21265 leader_election.cc:288] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:21.786947 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:21.787832 18293 raft_consensus.cc:2454] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:39:21.790174 18190 raft_consensus.cc:1261] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260807 03:39:21.706813 21263 consensus_queue.cc:1035] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.122s
I20260807 03:39:21.791195 18141 leader_election.cc:302] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:21.793413 21383 raft_consensus.cc:2781] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:39:21.793831 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:21.794333 21408 raft_consensus.cc:2781] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader election won for term 8
I20260807 03:39:21.800968 18027 leader_election.cc:302] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:21.774906 21239 leader_election.cc:288] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:21.814623 18187 raft_consensus.cc:2454] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 7.
I20260807 03:39:21.798530 21383 raft_consensus.cc:482] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:21.822865 18072 raft_consensus.cc:1261] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260807 03:39:21.837736 18072 pending_rounds.cc:76] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d: Aborting all ops after (but not including) 2
I20260807 03:39:21.838047 18072 pending_rounds.cc:97] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260807 03:39:21.838338 18072 raft_consensus.cc:2866] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 03:39:21.824432 21411 raft_consensus.cc:2781] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:21.826530 18249 leader_election.cc:302] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:21.802706 20583 raft_consensus.cc:2726] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:21.809163 21408 raft_consensus.cc:686] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 8 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:21.858047 18072 raft_consensus.cc:1261] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260807 03:39:21.841566 21411 raft_consensus.cc:482] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:21.842299 21383 raft_consensus.cc:3037] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:21.850059 20798 raft_consensus.cc:2726] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021) with higher term. Message: Invalid argument: T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 4. Current term is 5.
I20260807 03:39:21.849270 21263 raft_consensus.cc:2781] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 03:39:21.878098 18140 leader_election.cc:302] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:21.882586 18140 leader_election.cc:302] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:21.883916 18140 leader_election.cc:302] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:21.886752 18140 leader_election.cc:302] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:21.887625 21409 raft_consensus.cc:2726] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:21.888687 21263 raft_consensus.cc:686] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
W20260807 03:39:21.891208 21419 log.cc:924] Time spent T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.107s user 0.000s sys 0.001s
W20260807 03:39:21.891855 21420 log.cc:924] Time spent T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.088s user 0.000s sys 0.000s
I20260807 03:39:21.892297 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:21.924564 18293 raft_consensus.cc:2454] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 9.
I20260807 03:39:21.876368 21411 raft_consensus.cc:3037] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:21.887472 21365 raft_consensus.cc:2726] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:21.887214 21366 raft_consensus.cc:2726] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:21.956317 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:21.957154 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:21.957612 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:21.958911 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:21.999886 18080 raft_consensus.cc:2454] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 7.
I20260807 03:39:22.000962 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:21.964713 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 78047.6us
I20260807 03:39:22.002310 18140 leader_election.cc:302] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:21.982141 18081 raft_consensus.cc:2454] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:21.884974 20917 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:21.982035 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:21.998593 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 73351.6us
I20260807 03:39:22.017752 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:21.907439 21384 raft_consensus.cc:2726] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:22.025558 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:22.033902 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:22.035147 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:21.992367 21413 log.cc:924] Time spent T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.273s user 0.001s sys 0.001s
I20260807 03:39:22.053689 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:22.051994 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:22.054351 18075 raft_consensus.cc:3037] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Advancing to term 8
W20260807 03:39:22.055503 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 53350.4us
I20260807 03:39:22.056252 18140 leader_election.cc:302] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
W20260807 03:39:22.060463 21404 log.cc:924] Time spent T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.524s user 0.006s sys 0.009s
I20260807 03:39:22.061612 21277 raft_consensus.cc:482] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:22.062259 21277 raft_consensus.cc:504] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:22.082116 20917 raft_consensus.cc:2741] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260807 03:39:22.083070 21366 raft_consensus.cc:2741] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260807 03:39:22.009343 20822 raft_consensus.cc:504] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:22.085595 20822 leader_election.cc:288] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
W20260807 03:39:22.023026 21418 log.cc:924] Time spent T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.272s user 0.001s sys 0.001s
I20260807 03:39:22.083652 18140 leader_election.cc:302] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:22.115942 18188 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44660 (request call id 1515) took 1158 ms. Trace:
I20260807 03:39:22.105177 18185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44660 (request call id 1514) took 1179 ms. Trace:
I20260807 03:39:22.120311 18191 raft_consensus.cc:1261] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260807 03:39:22.117017 18185 rpcz_store.cc:276] 0807 03:39:20.926187 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:20.927669 (+ 1482us) service_pool.cc:227] Handling call
0807 03:39:20.944754 (+ 17085us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:21.030915 (+ 86161us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:21.061167 (+ 30252us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:39:21.063164 (+ 1997us) log.cc:842] Serialized 26 byte log entry
0807 03:39:21.306038 (+242874us) raft_consensus.cc:1631] Marking committed up to 2
0807 03:39:21.306073 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:21.306143 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:22.059789 (+753646us) spinlock_profiling.cc:243] Waited 3.15 ms on lock 0x7b44000232cc. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f1a9a95 00007fe98f1aa4b2 00007fe991e52511 00007fe991e4de1a 00007fe991e41bee 00007fe991e40f36 00007fe991e40b58 00007fe991e4a551 00007fe991e4a509 00007fe991e4a499
0807 03:39:22.086830 (+ 27041us) raft_consensus.cc:1671] finished
0807 03:39:22.086857 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:22.105117 (+ 18260us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":291270,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":8821,"spinlock_wait_cycles":7235840,"thread_start_us":268494,"threads_started":2,"wal-append.queue_time_us":390068}
I20260807 03:39:22.116628 18188 rpcz_store.cc:276] 0807 03:39:20.957120 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:20.957246 (+ 126us) service_pool.cc:227] Handling call
0807 03:39:20.957558 (+ 312us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:21.426553 (+468995us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:21.426597 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:39:21.449028 (+ 22431us) log.cc:842] Serialized 26 byte log entry
0807 03:39:21.585666 (+136638us) raft_consensus.cc:1631] Marking committed up to 2
0807 03:39:21.585699 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:22.103831 (+518132us) spinlock_profiling.cc:243] Waited 27.6 ms on lock 0x7b5400151154. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:39:22.103876 (+ 45us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:22.108007 (+ 4131us) raft_consensus.cc:1671] finished
0807 03:39:22.108028 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:22.115871 (+ 7843us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":254611,"raft.queue_time_us":264792,"raft.run_cpu_time_us":75,"raft.run_wall_time_us":75,"spinlock_wait_cycles":63402368,"thread_start_us":271055,"threads_started":2,"wal-append.queue_time_us":253232}
W20260807 03:39:22.131132 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 47596.3us
I20260807 03:39:22.132692 18140 leader_election.cc:302] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:22.134387 18140 leader_election.cc:302] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:22.156745 18140 leader_election.cc:302] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
W20260807 03:39:22.169072 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 408511us
I20260807 03:39:22.137995 21319 raft_consensus.cc:2726] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 03:39:22.173840 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:22.176090 18073 raft_consensus.cc:2379] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 5: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:22.137837 21115 raft_consensus.cc:2726] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:21.996663 21408 consensus_queue.cc:237] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:22.176283 18079 raft_consensus.cc:1261] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 03:39:22.183521 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:22.165905 21384 raft_consensus.cc:2781] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260807 03:39:22.219300 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 62659.4us
I20260807 03:39:22.222987 18140 leader_election.cc:302] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:22.220136 21115 raft_consensus.cc:2726] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 03:39:22.229674 18140 leader_election.cc:302] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:22.231045 18140 leader_election.cc:302] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:22.232435 18140 leader_election.cc:302] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:22.123512 21263 consensus_queue.cc:237] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:22.181414 21277 leader_election.cc:288] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:22.236876 18140 leader_election.cc:302] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:22.221951 21384 raft_consensus.cc:482] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:22.249001 17920 catalog_manager.cc:5495] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 8 to 9, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 9 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:39:22.233438 21319 raft_consensus.cc:2726] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:22.234004 21271 raft_consensus.cc:2726] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 03:39:22.234539 21239 raft_consensus.cc:2726] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 03:39:22.254745 20277 raft_consensus.cc:482] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:22.240841 21115 raft_consensus.cc:2726] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:22.265043 20277 raft_consensus.cc:504] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:22.269311 18140 leader_election.cc:302] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:22.251701 21384 raft_consensus.cc:3037] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:22.249981 21273 raft_consensus.cc:2726] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:22.283687 20277 leader_election.cc:288] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:22.288837 21239 raft_consensus.cc:2726] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:22.290194 18140 leader_election.cc:302] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:22.291352 21271 raft_consensus.cc:2726] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260807 03:39:22.297283 21423 log.cc:924] Time spent T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.426s user 0.000s sys 0.002s
I20260807 03:39:22.300590 17915 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39494 (request call id 136) took 3768 ms. Trace:
I20260807 03:39:22.306608 17920 catalog_manager.cc:5495] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 2 to 4, leader changed from d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 4 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:22.309644 18184 raft_consensus.cc:1261] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
W20260807 03:39:22.324847 21439 log.cc:924] Time spent T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
W20260807 03:39:22.326237 21435 log.cc:924] Time spent T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
W20260807 03:39:22.329912 21416 log.cc:924] Time spent T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.604s user 0.008s sys 0.006s
I20260807 03:39:22.265553 20822 raft_consensus.cc:482] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:22.334134 20822 raft_consensus.cc:504] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:22.336536 20822 leader_election.cc:288] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:22.334252 21329 consensus_queue.cc:1035] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260807 03:39:22.354485 21223 consensus_meta.cc:287] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.604s user 0.009s sys 0.000s
I20260807 03:39:22.359344 21260 raft_consensus.cc:482] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:22.416536 21260 raft_consensus.cc:504] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:22.419061 21260 leader_election.cc:288] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:22.420470 21223 raft_consensus.cc:504] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:22.433455 18295 raft_consensus.cc:1226] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 6. Current term is 7. Ops: []
W20260807 03:39:22.377869 21060 consensus_meta.cc:287] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.765s user 0.011s sys 0.000s
I20260807 03:39:22.435285 21060 raft_consensus.cc:504] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:22.436698 18297 raft_consensus.cc:2454] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 7.
I20260807 03:39:22.442004 21060 leader_election.cc:288] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:22.320722 17920 catalog_manager.cc:5495] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:22.444262 17920 catalog_manager.cc:5495] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 1 to 5. New cstate: current_term: 5 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:39:22.446295 17920 catalog_manager.cc:5495] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 1 to 3, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 3 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:39:22.448015 21273 raft_consensus.cc:482] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:22.448746 21273 raft_consensus.cc:504] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:22.451047 21273 leader_election.cc:288] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:22.451938 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:22.301069 17915 rpcz_store.cc:276] 0807 03:39:18.532702 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:18.532844 (+ 142us) service_pool.cc:227] Handling call
0807 03:39:22.300499 (+3767655us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 03:39:19.061547 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 03:39:19.062182 (+ 635us) write_op.cc:432] Acquiring schema lock in shared mode
0807 03:39:19.062238 (+ 56us) write_op.cc:435] Acquired schema lock
0807 03:39:19.062250 (+ 12us) tablet.cc:660] Decoding operations
0807 03:39:19.065605 (+ 3355us) write_op.cc:620] Acquiring the partition lock for write op
0807 03:39:19.065699 (+ 94us) write_op.cc:641] Partition lock acquired for write op
0807 03:39:19.065740 (+ 41us) tablet.cc:683] Acquiring locks for 18 operations
0807 03:39:19.067208 (+ 1468us) tablet.cc:699] Row locks acquired
0807 03:39:19.067246 (+ 38us) write_op.cc:260] PREPARE: finished
0807 03:39:19.067410 (+ 164us) write_op.cc:270] Start()
0807 03:39:19.067553 (+ 143us) write_op.cc:276] Timestamp: P: 1786073959067350 usec, L: 0
0807 03:39:19.067592 (+ 39us) op_driver.cc:331] REPLICATION: starting
0807 03:39:19.085358 (+ 17766us) log.cc:842] Serialized 7187 byte log entry
0807 03:39:19.284834 (+199476us) op_driver.cc:444] REPLICATION: finished
0807 03:39:19.430366 (+145532us) write_op.cc:301] APPLY: starting
0807 03:39:19.430484 (+ 118us) tablet.cc:1365] starting BulkCheckPresence
0807 03:39:19.430855 (+ 371us) tablet.cc:1368] finished BulkCheckPresence
0807 03:39:19.431010 (+ 155us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 03:39:19.433573 (+ 2563us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 03:39:19.434077 (+ 504us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=18
0807 03:39:19.434153 (+ 76us) write_op.cc:312] APPLY: finished
0807 03:39:19.434729 (+ 576us) log.cc:842] Serialized 124 byte log entry
0807 03:39:19.434997 (+ 268us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:39:19.435233 (+ 236us) write_op.cc:454] Released schema lock
0807 03:39:19.435339 (+ 106us) write_op.cc:341] FINISH: Updating metrics
0807 03:39:19.435697 (+ 358us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:39:19.435738 (+ 41us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":48235,"reported_tablets":41,"child_traces":[["op",{"apply.queue_time_us":139670,"apply.run_cpu_time_us":5998,"apply.run_wall_time_us":11218,"num_ops":18,"prepare.queue_time_us":105014,"prepare.run_cpu_time_us":28440,"prepare.run_wall_time_us":83934,"raft.queue_time_us":76981,"raft.run_cpu_time_us":22229,"raft.run_wall_time_us":171799,"replication_time_us":194644,"thread_start_us":311758,"threads_started":4,"wal-append.queue_time_us":58192,"wal-append.run_cpu_time_us":17722,"wal-append.run_wall_time_us":1292552}]]}
I20260807 03:39:22.374540 18305 raft_consensus.cc:1261] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260807 03:39:22.455977 18311 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.581s user 0.042s sys 0.061s
W20260807 03:39:22.456333 18311 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.582s user 0.042s sys 0.061s
I20260807 03:39:22.407253 18075 raft_consensus.cc:2454] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 8.
W20260807 03:39:22.457664 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 476201us
I20260807 03:39:22.458307 18141 leader_election.cc:302] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
W20260807 03:39:22.415930 21383 consensus_meta.cc:287] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 0.530s user 0.010s sys 0.000s
I20260807 03:39:22.461117 17920 catalog_manager.cc:5495] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 5 to 7, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 7 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:22.486701 17920 catalog_manager.cc:5495] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:22.491583 17920 catalog_manager.cc:5495] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 3 to 6, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 6 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:39:22.489529 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:22.504244 18295 raft_consensus.cc:1261] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260807 03:39:22.473433 21383 raft_consensus.cc:504] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:22.478264 21411 raft_consensus.cc:504] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:22.522843 17920 catalog_manager.cc:5495] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 1 to 2, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 2 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:22.526940 21411 leader_election.cc:288] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:22.527168 21458 log.cc:924] Time spent T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.205s user 0.000s sys 0.001s
I20260807 03:39:22.527498 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:22.531023 21383 leader_election.cc:288] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:22.531731 18189 raft_consensus.cc:1226] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 4. Current term is 5. Ops: []
I20260807 03:39:22.532445 20798 consensus_queue.cc:1035] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260807 03:39:22.532598 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:22.533387 18080 raft_consensus.cc:3037] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:22.519351 21329 consensus_queue.cc:1035] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20260807 03:39:22.515954 21224 consensus_queue.cc:1046] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.033s
I20260807 03:39:22.549893 21224 raft_consensus.cc:3032] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 6 LEADER]: Stepping down as leader of term 6
I20260807 03:39:22.550258 21224 raft_consensus.cc:729] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:22.551035 21224 consensus_queue.cc:260] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:22.552702 18080 raft_consensus.cc:2454] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 6.
I20260807 03:39:22.554018 21384 raft_consensus.cc:504] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:22.555601 21223 leader_election.cc:288] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:22.556948 21384 leader_election.cc:288] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:22.558164 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:22.558745 18072 raft_consensus.cc:3037] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:22.559244 21224 raft_consensus.cc:3037] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:22.565337 18140 leader_election.cc:302] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
W20260807 03:39:22.565496 21438 log.cc:924] Time spent T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.225s user 0.002s sys 0.000s
I20260807 03:39:22.566730 21409 raft_consensus.cc:2781] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader election won for term 6
W20260807 03:39:22.563815 21445 log.cc:924] Time spent T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.282s user 0.001s sys 0.001s
I20260807 03:39:22.567245 21409 raft_consensus.cc:686] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:22.568177 21409 consensus_queue.cc:237] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:22.581209 21329 consensus_queue.cc:1046] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:39:22.582150 21329 raft_consensus.cc:3032] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 LEADER]: Stepping down as leader of term 4
I20260807 03:39:22.582571 21329 raft_consensus.cc:729] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:22.583503 21329 consensus_queue.cc:260] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:22.585234 21329 raft_consensus.cc:3037] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:22.587971 18072 raft_consensus.cc:2454] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 8.
I20260807 03:39:22.589167 18140 leader_election.cc:302] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:22.590034 21383 raft_consensus.cc:2781] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader election won for term 8
I20260807 03:39:22.590561 21383 raft_consensus.cc:686] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:22.591502 21383 consensus_queue.cc:237] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:22.608719 18142 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.626s user 0.047s sys 0.080s
W20260807 03:39:22.609190 18142 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.627s user 0.047s sys 0.080s
I20260807 03:39:22.610265 21223 consensus_queue.cc:1035] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260807 03:39:22.612134 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 337954us
I20260807 03:39:22.612869 18248 leader_election.cc:302] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:22.614215 21461 raft_consensus.cc:2726] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:22.614511 21060 consensus_queue.cc:1035] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260807 03:39:22.616600 19239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 319946us
I20260807 03:39:22.626158 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:22.633538 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:22.637154 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:22.638204 18193 raft_consensus.cc:1261] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 03:39:22.639665 18191 raft_consensus.cc:1261] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260807 03:39:22.642691 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:22.667289 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:22.670414 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:22.671113 18311 raft_consensus.cc:3037] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:22.675587 18248 leader_election.cc:302] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:22.676465 21060 raft_consensus.cc:2726] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260807 03:39:22.678753 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:22.679544 18305 raft_consensus.cc:3037] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:22.690035 18305 raft_consensus.cc:2454] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 8.
I20260807 03:39:22.690250 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:22.691367 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:22.691948 18311 raft_consensus.cc:2454] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 6.
I20260807 03:39:22.693054 21115 raft_consensus.cc:2781] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260807 03:39:22.694419 21115 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:22.694795 21115 raft_consensus.cc:3037] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Advancing to term 10
I20260807 03:39:22.696012 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:22.703012 21115 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:22.719306 18141 leader_election.cc:302] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:22.722299 18141 leader_election.cc:302] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:22.746078 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:22.717630 21060 consensus_queue.cc:1035] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.014s
I20260807 03:39:22.749017 18072 raft_consensus.cc:1261] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260807 03:39:22.749588 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:22.752108 18075 raft_consensus.cc:2362] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4. Re-sending same reply.
I20260807 03:39:22.758284 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:22.758927 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:22.759351 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:22.758944 18079 raft_consensus.cc:2354] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 7. Current term is 8.
I20260807 03:39:22.762979 18248 leader_election.cc:398] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Vote denied by peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163) with higher term. Message: Invalid argument: T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 7. Current term is 8.
I20260807 03:39:22.763386 18248 leader_election.cc:401] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260807 03:39:22.775457 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:22.777648 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:39:22.782109 20277 consensus_queue.cc:1035] T 922e1d3cde8f476ea7780363c6b8693c P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 03:39:22.731516 21319 raft_consensus.cc:2726] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 03:39:22.731516 20653 raft_consensus.cc:2726] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260807 03:39:22.799943 18080 raft_consensus.cc:1261] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260807 03:39:22.802922 18249 leader_election.cc:302] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:22.792212 21336 raft_consensus.cc:2726] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163) with higher term. Message: Invalid argument: T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 7. Current term is 8.
I20260807 03:39:22.740566 20816 consensus_queue.cc:1035] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.062s
I20260807 03:39:22.811023 21263 raft_consensus.cc:2726] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 03:39:22.751103 21166 consensus_queue.cc:1035] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.058s
I20260807 03:39:22.837926 18081 raft_consensus.cc:2454] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 7.
I20260807 03:39:22.838837 17912 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35098 (request call id 28) took 1896 ms. Trace:
I20260807 03:39:22.839217 17912 rpcz_store.cc:276] 0807 03:39:20.942303 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:20.943076 (+ 773us) service_pool.cc:227] Handling call
0807 03:39:22.838750 (+1895674us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 03:39:22.253099 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 03:39:22.329889 (+ 76790us) write_op.cc:432] Acquiring schema lock in shared mode
0807 03:39:22.329939 (+ 50us) write_op.cc:435] Acquired schema lock
0807 03:39:22.329953 (+ 14us) tablet.cc:660] Decoding operations
0807 03:39:22.330295 (+ 342us) write_op.cc:620] Acquiring the partition lock for write op
0807 03:39:22.330377 (+ 82us) write_op.cc:641] Partition lock acquired for write op
0807 03:39:22.330414 (+ 37us) tablet.cc:683] Acquiring locks for 1 operations
0807 03:39:22.330656 (+ 242us) tablet.cc:699] Row locks acquired
0807 03:39:22.330676 (+ 20us) write_op.cc:260] PREPARE: finished
0807 03:39:22.330840 (+ 164us) write_op.cc:270] Start()
0807 03:39:22.330962 (+ 122us) write_op.cc:276] Timestamp: P: 1786073962330788 usec, L: 0
0807 03:39:22.330985 (+ 23us) op_driver.cc:331] REPLICATION: starting
0807 03:39:22.331682 (+ 697us) log.cc:842] Serialized 602 byte log entry
0807 03:39:22.578575 (+246893us) spinlock_profiling.cc:243] Waited 125 ms on lock 0x7b44000232cc. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f1a9a95 00007fe98f1aa4b2 00007fe991e52511 00007fe991e4de1a 00007fe98f2795fe 00007fe9980b0a39 00007fe98f2968bb 00007fe98f294cbf 00007fe98f296e22 00007fe991e405bb
0807 03:39:22.797088 (+218513us) op_driver.cc:444] REPLICATION: finished
0807 03:39:22.815805 (+ 18717us) write_op.cc:301] APPLY: starting
0807 03:39:22.819646 (+ 3841us) tablet.cc:1365] starting BulkCheckPresence
0807 03:39:22.823725 (+ 4079us) tablet.cc:1368] finished BulkCheckPresence
0807 03:39:22.825250 (+ 1525us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 03:39:22.829502 (+ 4252us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 03:39:22.829677 (+ 175us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0807 03:39:22.829724 (+ 47us) write_op.cc:312] APPLY: finished
0807 03:39:22.829914 (+ 190us) log.cc:842] Serialized 22 byte log entry
0807 03:39:22.830071 (+ 157us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:39:22.830158 (+ 87us) write_op.cc:454] Released schema lock
0807 03:39:22.830198 (+ 40us) write_op.cc:341] FINISH: Updating metrics
0807 03:39:22.830462 (+ 264us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:39:22.830492 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":645998,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":1167,"apply.run_cpu_time_us":1968,"apply.run_wall_time_us":21755,"num_ops":1,"prepare.queue_time_us":462470,"prepare.run_cpu_time_us":15883,"prepare.run_wall_time_us":489937,"raft.queue_time_us":70410,"raft.run_cpu_time_us":1318,"raft.run_wall_time_us":1315,"replication_time_us":465999,"spinlock_wait_cycles":288128768,"thread_start_us":798208,"threads_started":4,"wal-append.queue_time_us":393044}]]}
I20260807 03:39:22.896874 18248 leader_election.cc:302] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:22.896636 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:22.905929 21391 raft_consensus.cc:2781] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 03:39:22.899307 21373 raft_consensus.cc:482] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:22.913683 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:22.910382 21251 raft_consensus.cc:482] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:22.900902 21329 raft_consensus.cc:482] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:22.924168 21251 raft_consensus.cc:504] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:22.929638 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:22.930706 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:22.906193 21453 raft_consensus.cc:482] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:22.906073 21363 raft_consensus.cc:482] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:22.906159 21115 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:22.949025 18140 leader_election.cc:302] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:22.958766 21510 raft_consensus.cc:482] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:22.907647 20816 consensus_queue.cc:1035] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20260807 03:39:22.926128 21251 leader_election.cc:288] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:22.948611 21224 raft_consensus.cc:482] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:22.926471 21391 raft_consensus.cc:686] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
W20260807 03:39:22.980284 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 172733us
I20260807 03:39:22.959273 21510 raft_consensus.cc:504] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:22.975310 21492 log.cc:924] Time spent T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.078s user 0.000s sys 0.001s
I20260807 03:39:22.984861 18249 leader_election.cc:302] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:22.962584 21408 raft_consensus.cc:2726] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260807 03:39:22.989828 18249 leader_election.cc:302] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:22.929212 21329 raft_consensus.cc:504] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:23.015614 18249 leader_election.cc:302] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
W20260807 03:39:23.017958 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 57423.2us
I20260807 03:39:22.926900 21373 raft_consensus.cc:504] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:22.991067 21277 raft_consensus.cc:2726] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:22.960304 21363 raft_consensus.cc:504] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:23.021636 21211 consensus_queue.cc:1035] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
I20260807 03:39:23.014685 21166 raft_consensus.cc:2726] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:23.023697 20822 raft_consensus.cc:2726] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260807 03:39:23.045814 19240 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 48685.4us
I20260807 03:39:23.048676 18072 raft_consensus.cc:1261] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260807 03:39:22.979027 21224 raft_consensus.cc:504] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:22.963406 21453 raft_consensus.cc:504] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:23.073987 18027 leader_election.cc:302] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
W20260807 03:39:23.078388 21490 log.cc:924] Time spent T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331: Append to log took a long time: real 0.183s user 0.001s sys 0.000s
I20260807 03:39:23.079443 18070 raft_consensus.cc:1261] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260807 03:39:23.080410 18071 raft_consensus.cc:1261] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 03:39:23.080139 21391 consensus_queue.cc:237] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:23.057216 19239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 97386.1us
I20260807 03:39:23.083783 18183 raft_consensus.cc:1226] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 9. Current term is 10. Ops: []
I20260807 03:39:23.086850 21367 raft_consensus.cc:2726] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 03:39:23.089457 21383 consensus_queue.cc:1035] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 03:39:23.090337 21329 leader_election.cc:288] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:23.069877 18295 consensus_queue.cc:237] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:39:23.090871 21408 consensus_queue.cc:1035] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.008s
I20260807 03:39:23.101959 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:23.117538 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 92974.7us
I20260807 03:39:23.125567 21224 leader_election.cc:288] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:23.127621 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:23.129817 21043 raft_consensus.cc:482] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:23.130858 18300 raft_consensus.cc:1261] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260807 03:39:23.130380 21043 raft_consensus.cc:504] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:23.150590 21043 leader_election.cc:288] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:23.140795 21329 raft_consensus.cc:482] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:23.151526 21329 raft_consensus.cc:504] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:23.153924 21329 leader_election.cc:288] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:23.157132 19292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:23.160226 21516 consensus_queue.cc:1046] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260807 03:39:23.160600 20653 consensus_queue.cc:1035] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 03:39:23.165516 21510 leader_election.cc:288] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:23.167682 21453 leader_election.cc:288] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:23.169034 21373 leader_election.cc:288] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:23.181538 18079 raft_consensus.cc:1261] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
W20260807 03:39:23.182969 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 144007us
I20260807 03:39:23.189985 18311 raft_consensus.cc:1261] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 03:39:23.185971 21384 raft_consensus.cc:482] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:23.182569 21508 consensus_queue.cc:1035] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.007s
W20260807 03:39:23.346125 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 121740us
W20260807 03:39:23.364760 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 159050us
I20260807 03:39:23.396662 18249 leader_election.cc:302] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:23.195307 21383 consensus_queue.cc:1035] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.153s
I20260807 03:39:23.212487 21384 raft_consensus.cc:504] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:23.192866 21260 consensus_queue.cc:1035] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.156s
I20260807 03:39:23.410494 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:23.412272 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:39:23.407351 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 183991us
I20260807 03:39:23.414963 20277 raft_consensus.cc:2726] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 03:39:23.415401 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ca0223f47b46fd955704eba5259370" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:23.416342 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:23.416887 18183 raft_consensus.cc:2454] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 4.
I20260807 03:39:23.204039 21409 consensus_queue.cc:1035] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.198s
I20260807 03:39:23.423899 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:23.426312 18295 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314"
dest_uuid: "66a36e1ac8e945e893668e282d394c49"
mode: GRACEFUL
new_leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:58736
I20260807 03:39:23.426990 18295 raft_consensus.cc:595] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Received request to transfer leadership to TS d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:23.427325 18295 raft_consensus.cc:603] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260807 03:39:23.411062 21509 log.cc:924] Time spent T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
W20260807 03:39:23.409389 21528 log.cc:924] Time spent T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20260807 03:39:23.451323 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:23.443837 21327 raft_consensus.cc:482] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:23.450179 21408 raft_consensus.cc:482] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:23.449985 21361 raft_consensus.cc:482] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:23.461102 21363 leader_election.cc:288] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:23.457576 18027 leader_election.cc:302] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:23.473361 21367 raft_consensus.cc:3032] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 9 LEADER]: Stepping down as leader of term 9
I20260807 03:39:23.440850 21228 raft_consensus.cc:482] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
W20260807 03:39:23.458017 21531 log.cc:924] Time spent T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260807 03:39:23.475287 21384 leader_election.cc:288] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:23.481544 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:23.487676 18027 leader_election.cc:302] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:23.490736 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:23.491919 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:23.499614 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:23.496831 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 81965.3us
I20260807 03:39:23.494990 21247 raft_consensus.cc:2726] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260807 03:39:23.512122 21543 log.cc:924] Time spent T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.052s user 0.001s sys 0.000s
I20260807 03:39:23.485837 21367 raft_consensus.cc:729] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
W20260807 03:39:23.517889 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 103318us
I20260807 03:39:23.519938 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:23.507526 21363 raft_consensus.cc:2781] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:23.480271 21327 raft_consensus.cc:504] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:23.535007 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ca0223f47b46fd955704eba5259370" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:23.536932 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:23.538177 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:23.485162 21361 raft_consensus.cc:504] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:23.544229 21542 log.cc:924] Time spent T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.078s user 0.000s sys 0.001s
I20260807 03:39:23.546074 21361 leader_election.cc:288] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:23.497430 21228 raft_consensus.cc:504] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:23.549024 18027 leader_election.cc:302] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:23.552047 18180 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6129789267024fac8a337070c10a472a"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
mode: GRACEFUL
new_leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:34758
I20260807 03:39:23.482761 21408 raft_consensus.cc:504] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:23.555403 21408 leader_election.cc:288] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:23.555774 21327 leader_election.cc:288] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:23.556198 18180 raft_consensus.cc:595] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Received request to transfer leadership to TS d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:23.532903 21363 raft_consensus.cc:482] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:23.556754 18180 raft_consensus.cc:603] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 03:39:23.556888 21363 raft_consensus.cc:3037] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:23.558926 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:23.560792 18027 leader_election.cc:302] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:23.581985 18027 leader_election.cc:302] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:23.590512 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:23.593942 18293 raft_consensus.cc:1261] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260807 03:39:23.597541 18073 raft_consensus.cc:1261] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260807 03:39:23.598052 18027 leader_election.cc:302] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:23.581784 21367 consensus_queue.cc:260] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:23.599431 18027 leader_election.cc:302] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:23.601804 18071 raft_consensus.cc:1261] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 03:39:23.640036 18027 leader_election.cc:302] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:23.611195 18193 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
mode: GRACEFUL
new_leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:34758
I20260807 03:39:23.569170 21361 raft_consensus.cc:2726] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:23.641151 18193 raft_consensus.cc:595] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Received request to transfer leadership to TS d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:23.641970 18193 raft_consensus.cc:603] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 03:39:23.592756 21517 raft_consensus.cc:2726] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:23.607312 21367 raft_consensus.cc:3037] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Advancing to term 10
I20260807 03:39:23.598129 21453 raft_consensus.cc:2726] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:23.651743 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:23.652350 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:23.652599 18187 raft_consensus.cc:2454] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 7.
I20260807 03:39:23.600296 21516 raft_consensus.cc:2726] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 03:39:23.653986 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:23.636747 21242 raft_consensus.cc:2726] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 03:39:23.602964 21228 leader_election.cc:288] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:23.645793 21329 raft_consensus.cc:2726] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260807 03:39:23.607507 21327 consensus_queue.cc:1159] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "66a36e1ac8e945e893668e282d394c49" responder_term: 9 status { last_received { term: 9 index: 7 } last_committed_idx: 7 last_received_current_leader { term: 9 index: 7 } }
I20260807 03:39:23.694371 18189 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
mode: GRACEFUL
new_leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:34758
I20260807 03:39:23.695159 18189 raft_consensus.cc:595] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Received request to transfer leadership to TS d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:23.695570 18189 raft_consensus.cc:603] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 03:39:23.751509 18190 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
mode: GRACEFUL
new_leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:34758
I20260807 03:39:23.752306 18190 raft_consensus.cc:595] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Received request to transfer leadership to TS d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:23.752717 18190 raft_consensus.cc:603] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 03:39:23.783028 21363 raft_consensus.cc:504] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:23.786654 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260807 03:39:23.798106 21391 consensus_queue.cc:1035] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 03:39:23.798874 21207 consensus_queue.cc:1035] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 03:39:23.809396 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:23.810184 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 158264us
I20260807 03:39:23.806823 21448 raft_consensus.cc:482] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:23.814664 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:23.818924 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:23.828536 18027 leader_election.cc:302] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:23.833109 21514 raft_consensus.cc:2781] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 03:39:23.833439 21514 raft_consensus.cc:482] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:23.833709 21514 raft_consensus.cc:3037] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:23.823808 21448 raft_consensus.cc:504] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:23.842046 21363 leader_election.cc:288] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:23.860176 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:23.860783 18190 raft_consensus.cc:3037] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
W20260807 03:39:23.867262 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 221409us
I20260807 03:39:23.873798 18190 raft_consensus.cc:2454] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 5.
I20260807 03:39:23.876811 18027 leader_election.cc:302] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:23.856832 21514 raft_consensus.cc:504] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:23.901587 18070 raft_consensus.cc:1226] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
W20260807 03:39:23.904860 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 91249.5us
I20260807 03:39:23.888871 21578 raft_consensus.cc:2781] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election won for term 5
I20260807 03:39:23.907915 18141 leader_election.cc:302] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
W20260807 03:39:23.914464 18246 consensus_peers.cc:487] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 -> Peer 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381): Couldn't send request to peer 5c9b5062bdfa46af9396f1a85025a331. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3433ce141bc54c5395e00fe40cf9bb69. This is attempt 1: this message will repeat every 5th retry.
I20260807 03:39:23.912925 21553 consensus_queue.cc:1035] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
W20260807 03:39:23.915572 18246 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.132:41381, user_credentials={real_user=slave}} blocked reactor thread for 93323us
I20260807 03:39:23.920477 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:23.910809 21546 raft_consensus.cc:2726] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:23.926218 21553 raft_consensus.cc:482] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:23.912652 21578 raft_consensus.cc:686] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:23.925503 18027 leader_election.cc:302] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:23.932232 18027 leader_election.cc:302] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:23.935053 21363 raft_consensus.cc:2726] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:23.933226 21251 raft_consensus.cc:2726] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:23.939117 21578 consensus_queue.cc:237] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:23.941231 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:23.954078 18187 raft_consensus.cc:1226] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 4. Current term is 5. Ops: []
I20260807 03:39:23.926752 21553 raft_consensus.cc:504] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:23.966774 18183 raft_consensus.cc:3037] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:23.971041 21553 leader_election.cc:288] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:23.974565 21448 leader_election.cc:288] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:23.982643 18183 raft_consensus.cc:1261] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20260807 03:39:23.984380 21562 log.cc:924] Time spent T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.287s user 0.000s sys 0.004s
I20260807 03:39:23.986543 21336 consensus_queue.cc:1046] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:39:23.987601 21336 raft_consensus.cc:3032] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 03:39:23.987998 21336 raft_consensus.cc:729] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:23.989564 21336 consensus_queue.cc:260] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:23.993875 18187 raft_consensus.cc:1261] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260807 03:39:24.001752 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:24.003501 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:24.004590 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 75619.1us
I20260807 03:39:24.016193 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:24.016796 18190 raft_consensus.cc:3037] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:24.019713 21514 leader_election.cc:288] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:24.024271 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:24.035879 21568 consensus_queue.cc:1035] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:39:24.036469 21580 raft_consensus.cc:2932] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:39:24.063052 18190 raft_consensus.cc:2454] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 8.
I20260807 03:39:24.074232 18071 raft_consensus.cc:1226] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 4. Current term is 5. Ops: []
I20260807 03:39:24.077440 18027 leader_election.cc:302] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:24.072450 18073 raft_consensus.cc:2932] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
W20260807 03:39:24.082628 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 68268.8us
I20260807 03:39:24.083509 18141 leader_election.cc:302] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:24.085667 19292 consensus_queue.cc:237] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:24.094641 21582 raft_consensus.cc:482] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:24.098368 18183 raft_consensus.cc:1226] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260807 03:39:24.095199 21582 raft_consensus.cc:504] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:24.112218 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:39:24.101586 21043 raft_consensus.cc:2781] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader election won for term 8
I20260807 03:39:24.131968 21043 raft_consensus.cc:686] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 8 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:24.133167 21043 consensus_queue.cc:237] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:24.110384 21530 consensus_queue.cc:1035] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20260807 03:39:24.131102 21582 leader_election.cc:288] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:24.103343 21408 raft_consensus.cc:2726] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260807 03:39:24.155376 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 53301.6us
W20260807 03:39:24.155570 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 47085.1us
I20260807 03:39:24.157968 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:24.161120 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:24.158339 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:24.161999 18070 raft_consensus.cc:2454] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:24.173638 21277 consensus_queue.cc:1046] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260807 03:39:24.180745 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 78089.8us
I20260807 03:39:24.186744 18248 leader_election.cc:302] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:24.184496 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:24.190259 18184 raft_consensus.cc:2379] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 5: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:24.170635 21555 raft_consensus.cc:482] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:24.193337 21546 raft_consensus.cc:482] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:24.211054 21166 consensus_queue.cc:1046] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:39:24.209185 21571 raft_consensus.cc:2781] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:24.193900 21580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:37176 (request call id 4) took 1372 ms. Trace:
I20260807 03:39:24.209998 18141 leader_election.cc:302] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:24.215394 18141 leader_election.cc:302] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:24.213207 18187 raft_consensus.cc:1261] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Refusing update from remote peer 5c9b5062bdfa46af9396f1a85025a331: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260807 03:39:24.202847 21555 raft_consensus.cc:504] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:24.214299 21571 raft_consensus.cc:482] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:24.211620 21546 raft_consensus.cc:504] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:24.218036 21408 raft_consensus.cc:2726] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:24.220702 18295 raft_consensus.cc:1261] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Refusing update from remote peer 5c9b5062bdfa46af9396f1a85025a331: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260807 03:39:24.231936 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:24.223289 21602 log.cc:924] Time spent T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.208s user 0.005s sys 0.000s
I20260807 03:39:24.234030 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:24.222332 21571 raft_consensus.cc:3037] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:24.222417 21555 leader_election.cc:288] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:24.244874 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:24.246066 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:24.246152 21520 consensus_queue.cc:1035] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260807 03:39:24.226008 21546 leader_election.cc:288] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:24.237015 21557 log.cc:924] Time spent T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.547s user 0.005s sys 0.000s
I20260807 03:39:24.251511 21063 consensus_queue.cc:1035] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260807 03:39:24.214571 21580 rpcz_store.cc:276] 0807 03:39:22.808180 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:22.808342 (+ 162us) service_pool.cc:227] Handling call
0807 03:39:23.063507 (+255165us) spinlock_profiling.cc:243] Waited 35.4 ms on lock 0x7b5400125d20. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe992154dc0 00007fe9921c5bd2 00007fe9921c4890 00007fe997101f39 00007fe9907ed704 00007fe9907ed692 00007fe9907ed5c1 00007fe9907ed53c 00007fe9907ec7f2 00007fe9905e994c
0807 03:39:23.147204 (+ 83697us) log.cc:842] Serialized 513 byte log entry
0807 03:39:23.475631 (+328427us) spinlock_profiling.cc:243] Waited 3.43 ms on lock 0x7b5400125d20. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe992150a4f 00007fe992139545 00007fe99213e771 00007fe99213e6f9 00007fe99213e689 00007fe99213e651 00007fe99213d81d 00007fe9984fb204 00007fe9984fb039 00007fe98f2976b6
0807 03:39:24.074830 (+599199us) spinlock_profiling.cc:243] Waited 2.65 ms on lock 0x7b5400125d20. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe992150a4f 00007fe992139545 00007fe99213e771 00007fe99213e6f9 00007fe99213e689 00007fe99213e651 00007fe99213d81d 00007fe9984fb204 00007fe9984fb039 00007fe98f2976b6
0807 03:39:24.092205 (+ 17375us) spinlock_profiling.cc:243] Waited 2.4 ms on lock 0x7b2c00108448. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe990ba2e11 00007fe990bb1ada 00007fe990bb1763 00007fe990bb1701 00007fe9921fbd62 00007fe9921fbf05 00007fe992150f91 00007fe992139545 00007fe99213e771 00007fe99213e6f9 00007fe99213e689
0807 03:39:24.177477 (+ 85272us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":208787,"raft.queue_time_us":193085,"raft.run_cpu_time_us":2419,"raft.run_wall_time_us":78762,"spinlock_wait_cycles":108760576,"thread_start_us":66634,"threads_started":1,"wal-append.queue_time_us":67056}
I20260807 03:39:24.260464 18141 leader_election.cc:302] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:24.261759 18141 leader_election.cc:302] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:24.263173 20653 raft_consensus.cc:2726] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 03:39:24.264946 21612 ts_tablet_manager.cc:927] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331: Initiating tablet copy from peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:24.261931 21447 raft_consensus.cc:2726] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:24.271235 21260 raft_consensus.cc:3032] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Stepping down as leader of term 7
I20260807 03:39:24.271620 21260 raft_consensus.cc:729] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:24.272419 21260 consensus_queue.cc:260] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:24.273834 21260 raft_consensus.cc:3037] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:24.252786 21336 raft_consensus.cc:3037] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:24.114588 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:39:24.290238 17920 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39492 (request call id 147) took 5733 ms. Trace:
I20260807 03:39:24.293337 18183 raft_consensus.cc:1261] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 03:39:24.298807 21223 consensus_queue.cc:1035] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:39:24.290586 17920 rpcz_store.cc:276] 0807 03:39:18.557179 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:18.557305 (+ 126us) service_pool.cc:227] Handling call
0807 03:39:24.290179 (+5732874us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 03:39:22.802593 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 03:39:22.863129 (+ 60536us) write_op.cc:432] Acquiring schema lock in shared mode
0807 03:39:22.863639 (+ 510us) write_op.cc:435] Acquired schema lock
0807 03:39:22.866090 (+ 2451us) tablet.cc:660] Decoding operations
0807 03:39:23.043037 (+176947us) write_op.cc:620] Acquiring the partition lock for write op
0807 03:39:23.050753 (+ 7716us) write_op.cc:641] Partition lock acquired for write op
0807 03:39:23.056294 (+ 5541us) tablet.cc:683] Acquiring locks for 9 operations
0807 03:39:23.096207 (+ 39913us) tablet.cc:699] Row locks acquired
0807 03:39:23.096227 (+ 20us) write_op.cc:260] PREPARE: finished
0807 03:39:23.096383 (+ 156us) write_op.cc:270] Start()
0807 03:39:23.096491 (+ 108us) write_op.cc:276] Timestamp: P: 1786073963096346 usec, L: 0
0807 03:39:23.096512 (+ 21us) op_driver.cc:331] REPLICATION: starting
0807 03:39:23.097073 (+ 561us) log.cc:842] Serialized 3668 byte log entry
0807 03:39:23.099288 (+ 2215us) op_driver.cc:444] REPLICATION: finished
0807 03:39:23.099800 (+ 512us) write_op.cc:301] APPLY: starting
0807 03:39:23.099899 (+ 99us) tablet.cc:1365] starting BulkCheckPresence
0807 03:39:23.101593 (+ 1694us) tablet.cc:1368] finished BulkCheckPresence
0807 03:39:23.132315 (+ 30722us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 03:39:23.170653 (+ 38338us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 03:39:23.170943 (+ 290us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=9
0807 03:39:23.171008 (+ 65us) write_op.cc:312] APPLY: finished
0807 03:39:23.171331 (+ 323us) log.cc:842] Serialized 70 byte log entry
0807 03:39:23.171510 (+ 179us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:39:23.171637 (+ 127us) write_op.cc:454] Released schema lock
0807 03:39:23.171683 (+ 46us) write_op.cc:341] FINISH: Updating metrics
0807 03:39:23.171958 (+ 275us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:39:23.171988 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1126555,"reported_tablets":35,"child_traces":[["op",{"apply.queue_time_us":253,"apply.run_cpu_time_us":3700,"apply.run_wall_time_us":85831,"num_ops":9,"prepare.queue_time_us":204748,"prepare.run_cpu_time_us":5332,"prepare.run_wall_time_us":304379,"replication_time_us":2693}]]}
I20260807 03:39:24.329979 21612 tablet_copy_client.cc:316] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Beginning tablet copy session from remote peer at address 127.15.198.131:33021
I20260807 03:39:24.345419 21571 raft_consensus.cc:504] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:24.347638 21571 leader_election.cc:288] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:24.349512 17914 catalog_manager.cc:5495] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 3 to 8, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 8 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:39:24.351495 21588 raft_consensus.cc:2726] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 03:39:24.361562 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:24.362602 18191 raft_consensus.cc:2379] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 5: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:24.353008 17914 catalog_manager.cc:5495] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 3 to 5, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 5 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:24.507753 17914 catalog_manager.cc:5495] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 2 to 3, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 3 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:39:24.509004 21327 raft_consensus.cc:482] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:24.509572 21327 raft_consensus.cc:504] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:24.511462 21327 leader_election.cc:288] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:24.513842 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 174736us
I20260807 03:39:24.519677 18325 tablet_copy_service.cc:140] P 66a36e1ac8e945e893668e282d394c49: Received BeginTabletCopySession request for tablet 3433ce141bc54c5395e00fe40cf9bb69 from peer 5c9b5062bdfa46af9396f1a85025a331 ({username='slave'} at 127.0.0.1:37190)
I20260807 03:39:24.520361 18325 tablet_copy_service.cc:161] P 66a36e1ac8e945e893668e282d394c49: Beginning new tablet copy session on tablet 3433ce141bc54c5395e00fe40cf9bb69 from peer 5c9b5062bdfa46af9396f1a85025a331 at {username='slave'} at 127.0.0.1:37190: session id = 5c9b5062bdfa46af9396f1a85025a331-3433ce141bc54c5395e00fe40cf9bb69
W20260807 03:39:24.520869 18246 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.132:41381, user_credentials={real_user=slave}} blocked reactor thread for 166244us
I20260807 03:39:24.523149 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:24.530175 18082 raft_consensus.cc:2454] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:24.535549 18325 tablet_copy_source_session.cc:205] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 03:39:24.546483 21335 raft_consensus.cc:2932] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 [term 7 LEADER]: Committing config change with OpId 7.7: config changed from index 3 to 7, VOTER d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:24.559270 17897 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 9a59a83a64df475ea15dd572f84024bd with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260807 03:39:24.559500 18003 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica 3c4bceb836db4536a0f255fdf86abc02 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:24.562706 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:24.568360 18184 raft_consensus.cc:2932] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Committing config change with OpId 7.7: config changed from index 3 to 7, VOTER d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:24.572348 18190 raft_consensus.cc:1261] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 03:39:24.573298 21612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3433ce141bc54c5395e00fe40cf9bb69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:39:24.512652 17914 catalog_manager.cc:5495] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 3 to 6, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 6 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:39:24.576819 17914 catalog_manager.cc:5495] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 2 to 4, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 4 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:24.581533 18248 leader_election.cc:302] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:24.583333 21581 consensus_queue.cc:1035] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:39:24.592628 21223 raft_consensus.cc:2781] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 03:39:24.593155 21223 raft_consensus.cc:686] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:24.593976 21223 consensus_queue.cc:237] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:24.611356 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:24.612109 17914 catalog_manager.cc:5495] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 5 to 7, leader changed from d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 7 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:39:24.615301 17914 catalog_manager.cc:5495] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 3 to 4, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 4 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:24.619980 17914 catalog_manager.cc:5495] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 4 to 6, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 6 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:24.629438 17914 catalog_manager.cc:5495] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 1 to 3, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 3 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:24.633430 17914 catalog_manager.cc:5495] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 3 to 5, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 5 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:24.629310 18191 raft_consensus.cc:2932] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:39:24.635921 18312 raft_consensus.cc:1261] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 03:39:24.643151 18027 leader_election.cc:302] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:24.646679 21327 consensus_queue.cc:1035] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260807 03:39:24.657583 18003 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 6b66966247c147308a4b8488557e0bdf, TS 66a36e1ac8e945e893668e282d394c49 -> TS 5c9b5062bdfa46af9396f1a85025a331 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:24.658010 18003 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 6b66966247c147308a4b8488557e0bdf, TS 66a36e1ac8e945e893668e282d394c49 -> TS 5c9b5062bdfa46af9396f1a85025a331 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:24.682221 18311 raft_consensus.cc:2932] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Committing config change with OpId 7.7: config changed from index 3 to 7, VOTER d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
W20260807 03:39:24.685201 18003 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 60437eed8852484c88e84993e376670e, TS d9716e2bf06442f5b3b0e81087c53c4d -> TS 5c9b5062bdfa46af9396f1a85025a331 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:24.685562 18003 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 60437eed8852484c88e84993e376670e, TS d9716e2bf06442f5b3b0e81087c53c4d -> TS 5c9b5062bdfa46af9396f1a85025a331 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:24.664141 21224 raft_consensus.cc:2726] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:24.697293 18183 raft_consensus.cc:1261] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260807 03:39:24.707978 21043 consensus_queue.cc:1035] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260807 03:39:24.715761 18003 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet c702701ec4504375977f142ffc7f0be0, TS 66a36e1ac8e945e893668e282d394c49 -> TS 5c9b5062bdfa46af9396f1a85025a331 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:24.716161 18003 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet c702701ec4504375977f142ffc7f0be0, TS 66a36e1ac8e945e893668e282d394c49 -> TS 5c9b5062bdfa46af9396f1a85025a331 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:24.772652 18311 raft_consensus.cc:1261] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 4. (term mismatch)
I20260807 03:39:24.772972 18311 pending_rounds.cc:76] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49: Aborting all ops after (but not including) 3
I20260807 03:39:24.773159 18311 pending_rounds.cc:97] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49: Aborting uncommitted NO_OP operation due to leader change: 7.4
I20260807 03:39:24.773352 18311 raft_consensus.cc:2866] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 03:39:24.779892 21363 consensus_queue.cc:1035] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:39:24.804381 21612 tablet_copy_client.cc:791] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Starting download of 0 data blocks...
I20260807 03:39:24.805153 21612 tablet_copy_client.cc:655] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Starting download of 1 WAL segments...
I20260807 03:39:24.811458 21612 tablet_copy_client.cc:531] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 03:39:24.837141 17920 catalog_manager.cc:5495] T 9a59a83a64df475ea15dd572f84024bd P 5c9b5062bdfa46af9396f1a85025a331 reported cstate change: config changed from index 3 to 7, VOTER d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) evicted. New cstate: current_term: 7 leader_uuid: "5c9b5062bdfa46af9396f1a85025a331" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 03:39:24.843380 21612 tablet_bootstrap.cc:492] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331: Bootstrap starting.
I20260807 03:39:24.864778 18064 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a59a83a64df475ea15dd572f84024bd with delete_type TABLET_DATA_TOMBSTONED (TS d9716e2bf06442f5b3b0e81087c53c4d not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33238
I20260807 03:39:24.876291 17915 catalog_manager.cc:5495] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 5 to 6, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 6 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:24.884794 17914 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39510 (request call id 141) took 6324 ms. Trace:
I20260807 03:39:24.882679 17915 catalog_manager.cc:5495] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 4 to 6, leader changed from d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 6 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:39:24.885673 21645 tablet_replica.cc:330] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d: stopping tablet replica
I20260807 03:39:24.885249 17914 rpcz_store.cc:276] 0807 03:39:18.560374 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:18.561143 (+ 769us) service_pool.cc:227] Handling call
0807 03:39:24.884729 (+6323586us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 03:39:24.774262 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 03:39:24.776559 (+ 2297us) write_op.cc:432] Acquiring schema lock in shared mode
0807 03:39:24.776595 (+ 36us) write_op.cc:435] Acquired schema lock
0807 03:39:24.776613 (+ 18us) tablet.cc:660] Decoding operations
0807 03:39:24.787684 (+ 11071us) write_op.cc:620] Acquiring the partition lock for write op
0807 03:39:24.787763 (+ 79us) write_op.cc:641] Partition lock acquired for write op
0807 03:39:24.787787 (+ 24us) tablet.cc:683] Acquiring locks for 10 operations
0807 03:39:24.788536 (+ 749us) tablet.cc:699] Row locks acquired
0807 03:39:24.788561 (+ 25us) write_op.cc:260] PREPARE: finished
0807 03:39:24.788724 (+ 163us) write_op.cc:270] Start()
0807 03:39:24.788838 (+ 114us) write_op.cc:276] Timestamp: P: 1786073964788678 usec, L: 0
0807 03:39:24.788859 (+ 21us) op_driver.cc:331] REPLICATION: starting
0807 03:39:24.789454 (+ 595us) log.cc:842] Serialized 4063 byte log entry
0807 03:39:24.803810 (+ 14356us) op_driver.cc:444] REPLICATION: finished
0807 03:39:24.814070 (+ 10260us) write_op.cc:301] APPLY: starting
0807 03:39:24.814156 (+ 86us) tablet.cc:1365] starting BulkCheckPresence
0807 03:39:24.814371 (+ 215us) tablet.cc:1368] finished BulkCheckPresence
0807 03:39:24.814493 (+ 122us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 03:39:24.817911 (+ 3418us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 03:39:24.818129 (+ 218us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=10
0807 03:39:24.818169 (+ 40us) write_op.cc:312] APPLY: finished
0807 03:39:24.818435 (+ 266us) log.cc:842] Serialized 76 byte log entry
0807 03:39:24.818617 (+ 182us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:39:24.818750 (+ 133us) write_op.cc:454] Released schema lock
0807 03:39:24.818798 (+ 48us) write_op.cc:341] FINISH: Updating metrics
0807 03:39:24.856569 (+ 37771us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:39:24.856612 (+ 43us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":172,"reported_tablets":53,"child_traces":[["op",{"apply.queue_time_us":9979,"apply.run_cpu_time_us":3422,"apply.run_wall_time_us":43577,"num_ops":10,"prepare.queue_time_us":27758,"prepare.run_cpu_time_us":4364,"prepare.run_wall_time_us":37151,"raft.queue_time_us":1774,"raft.run_cpu_time_us":13139,"raft.run_wall_time_us":27617,"replication_time_us":14851,"thread_start_us":35477,"threads_started":4,"wal-append.queue_time_us":11000}]]}
I20260807 03:39:24.887935 17915 catalog_manager.cc:5495] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 4 to 6, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 6 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:39:24.890852 17915 catalog_manager.cc:5495] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:24.892306 17915 catalog_manager.cc:5495] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 3 to 4, leader changed from d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 4 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:24.894280 17915 catalog_manager.cc:5495] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:24.895648 17915 catalog_manager.cc:5495] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 1 to 5, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 5 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:24.896996 17915 catalog_manager.cc:5495] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 3 to 7. New cstate: current_term: 7 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:39:24.898463 17915 catalog_manager.cc:5495] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 2 to 3, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 3 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:24.901216 17915 catalog_manager.cc:5495] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 5 to 6, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 6 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:24.903107 17915 catalog_manager.cc:5495] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 3 to 4, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 4 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:24.905642 17915 catalog_manager.cc:5495] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:39:24.907263 17915 catalog_manager.cc:5495] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:24.908744 17915 catalog_manager.cc:5495] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:24.911101 17915 catalog_manager.cc:5495] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 3 to 5, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 5 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:24.913738 17915 catalog_manager.cc:5495] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:24.915127 17915 catalog_manager.cc:5495] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 4 to 5, leader changed from d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 5 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:24.916956 17915 catalog_manager.cc:5495] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:24.918421 17915 catalog_manager.cc:5495] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 3 to 4, leader changed from d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 4 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:24.921629 17915 catalog_manager.cc:5495] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 2 to 3, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 3 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:24.923202 17915 catalog_manager.cc:5495] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:39:24.924959 17915 catalog_manager.cc:5495] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:24.927958 17915 catalog_manager.cc:5495] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 3 to 7. New cstate: current_term: 7 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:24.934764 21645 raft_consensus.cc:2229] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Raft consensus shutting down.
I20260807 03:39:24.935743 21645 raft_consensus.cc:2258] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Raft consensus is shut down!
I20260807 03:39:24.938815 21645 ts_tablet_manager.cc:1905] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 03:39:24.967593 21645 ts_tablet_manager.cc:1918] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 7.6
I20260807 03:39:24.977407 21645 log.cc:1196] T 9a59a83a64df475ea15dd572f84024bd P d9716e2bf06442f5b3b0e81087c53c4d: Deleting WAL directory at /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786073845599630-16154-0/minicluster-data/ts-0-root/wals/9a59a83a64df475ea15dd572f84024bd
I20260807 03:39:24.980522 17898 catalog_manager.cc:4841] TS d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): tablet 9a59a83a64df475ea15dd572f84024bd (table test-workload [id=677ca051b50345e183d60556fc42291d]) successfully deleted
I20260807 03:39:24.985566 21612 tablet_bootstrap.cc:492] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 03:39:24.986617 21612 tablet_bootstrap.cc:492] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331: Bootstrap complete.
I20260807 03:39:24.987411 21612 ts_tablet_manager.cc:1397] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331: Time spent bootstrapping tablet: real 0.144s user 0.022s sys 0.005s
I20260807 03:39:24.997012 21612 raft_consensus.cc:348] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:39:24.998771 21612 raft_consensus.cc:729] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 5c9b5062bdfa46af9396f1a85025a331, State: Initialized, Role: LEARNER
I20260807 03:39:24.999439 21612 consensus_queue.cc:260] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:39:25.001709 21611 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:25.002249 21611 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:25.005748 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:25.006536 18080 raft_consensus.cc:2454] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 10.
I20260807 03:39:25.007416 21612 ts_tablet_manager.cc:1428] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331: Time spent starting tablet: real 0.020s user 0.007s sys 0.003s
I20260807 03:39:25.009452 18325 tablet_copy_service.cc:342] P 66a36e1ac8e945e893668e282d394c49: Request end of tablet copy session 5c9b5062bdfa46af9396f1a85025a331-3433ce141bc54c5395e00fe40cf9bb69 received from {username='slave'} at 127.0.0.1:37190
I20260807 03:39:25.009881 18325 tablet_copy_service.cc:433] P 66a36e1ac8e945e893668e282d394c49: ending tablet copy session 5c9b5062bdfa46af9396f1a85025a331-3433ce141bc54c5395e00fe40cf9bb69 on tablet 3433ce141bc54c5395e00fe40cf9bb69 with peer 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:39:25.035702 21571 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:25.036497 21571 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:25.039847 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:25.040768 18191 raft_consensus.cc:2379] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 10: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:25.046738 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:25.047348 18082 raft_consensus.cc:2454] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 10.
I20260807 03:39:25.051205 18248 leader_election.cc:302] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:25.055544 21321 raft_consensus.cc:2781] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260807 03:39:25.055958 21321 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:25.056342 21321 raft_consensus.cc:3037] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Advancing to term 10
I20260807 03:39:25.081820 18140 leader_election.cc:302] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:25.087205 21447 raft_consensus.cc:2781] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260807 03:39:25.093405 21447 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:25.093712 21447 raft_consensus.cc:3037] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Advancing to term 11
I20260807 03:39:25.096792 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:25.098306 21611 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:25.085866 21571 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:25.111377 21321 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:25.112340 18295 raft_consensus.cc:2454] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 10.
I20260807 03:39:25.114689 21321 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:25.116245 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:25.118846 21447 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:25.119463 17915 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39494 (request call id 137) took 1411 ms. Trace:
I20260807 03:39:25.120137 18183 raft_consensus.cc:2354] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 10. Current term is 11.
I20260807 03:39:25.122062 18249 leader_election.cc:398] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 election: Vote denied by peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057) with higher term. Message: Invalid argument: T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 10. Current term is 11.
I20260807 03:39:25.122463 18249 leader_election.cc:401] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20260807 03:39:25.119879 17915 rpcz_store.cc:276] 0807 03:39:23.709772 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:23.713842 (+ 4070us) service_pool.cc:227] Handling call
0807 03:39:25.119397 (+1405555us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 03:39:24.936555 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 03:39:24.937055 (+ 500us) write_op.cc:432] Acquiring schema lock in shared mode
0807 03:39:24.937092 (+ 37us) write_op.cc:435] Acquired schema lock
0807 03:39:24.937104 (+ 12us) tablet.cc:660] Decoding operations
0807 03:39:24.938797 (+ 1693us) write_op.cc:620] Acquiring the partition lock for write op
0807 03:39:24.938860 (+ 63us) write_op.cc:641] Partition lock acquired for write op
0807 03:39:24.938886 (+ 26us) tablet.cc:683] Acquiring locks for 23 operations
0807 03:39:24.940666 (+ 1780us) tablet.cc:699] Row locks acquired
0807 03:39:24.940688 (+ 22us) write_op.cc:260] PREPARE: finished
0807 03:39:24.940819 (+ 131us) write_op.cc:270] Start()
0807 03:39:24.940922 (+ 103us) write_op.cc:276] Timestamp: P: 1786073964940777 usec, L: 0
0807 03:39:24.940943 (+ 21us) op_driver.cc:331] REPLICATION: starting
0807 03:39:24.942391 (+ 1448us) log.cc:842] Serialized 9142 byte log entry
0807 03:39:24.948144 (+ 5753us) op_driver.cc:444] REPLICATION: finished
0807 03:39:24.950439 (+ 2295us) write_op.cc:301] APPLY: starting
0807 03:39:24.950511 (+ 72us) tablet.cc:1365] starting BulkCheckPresence
0807 03:39:24.951102 (+ 591us) tablet.cc:1368] finished BulkCheckPresence
0807 03:39:24.951212 (+ 110us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 03:39:24.953347 (+ 2135us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 03:39:24.953967 (+ 620us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=23
0807 03:39:24.954018 (+ 51us) write_op.cc:312] APPLY: finished
0807 03:39:24.954432 (+ 414us) log.cc:842] Serialized 157 byte log entry
0807 03:39:24.954599 (+ 167us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:39:24.954810 (+ 211us) write_op.cc:454] Released schema lock
0807 03:39:24.954873 (+ 63us) write_op.cc:341] FINISH: Updating metrics
0807 03:39:24.955151 (+ 278us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:39:24.955181 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":309691,"reported_tablets":42,"child_traces":[["op",{"apply.queue_time_us":2051,"apply.run_cpu_time_us":5263,"apply.run_wall_time_us":5311,"num_ops":23,"prepare.queue_time_us":114,"prepare.run_cpu_time_us":5326,"prepare.run_wall_time_us":6105,"replication_time_us":7098}]]}
I20260807 03:39:25.123498 21461 raft_consensus.cc:3037] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Advancing to term 11
I20260807 03:39:25.133646 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:25.143141 19292 raft_consensus.cc:1203] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3] Dedup: 3.3->[]
I20260807 03:39:25.144909 18073 raft_consensus.cc:1261] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 03:39:25.146247 18082 raft_consensus.cc:2454] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 10.
I20260807 03:39:25.150271 18183 raft_consensus.cc:1261] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 03:39:25.138692 17912 catalog_manager.cc:5495] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 4 to 5, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 5 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:25.152488 17912 catalog_manager.cc:5495] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:25.156525 17912 catalog_manager.cc:5495] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 5 to 8, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 8 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:25.158057 17912 catalog_manager.cc:5495] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 6 to 7, leader changed from d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 7 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:25.160382 17912 catalog_manager.cc:5495] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: config changed from index -1 to 3, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New cstate: current_term: 3 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:39:25.165079 17912 catalog_manager.cc:5495] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:25.158823 21221 consensus_queue.cc:1035] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.011s
I20260807 03:39:25.168082 20678 consensus_queue.cc:1035] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:39:25.173413 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:25.173995 18081 raft_consensus.cc:3037] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Advancing to term 11
I20260807 03:39:25.182966 18081 raft_consensus.cc:2454] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 11.
I20260807 03:39:25.187541 18140 leader_election.cc:302] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:25.176591 21447 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 11 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:25.191524 20653 raft_consensus.cc:2781] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 11 FOLLOWER]: Leader election won for term 11
I20260807 03:39:25.198161 20653 raft_consensus.cc:686] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 11 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:25.199051 20653 consensus_queue.cc:237] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:25.220467 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:25.303911 21461 raft_consensus.cc:2726] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057) with higher term. Message: Invalid argument: T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 10. Current term is 11.
I20260807 03:39:25.318125 18295 raft_consensus.cc:2454] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 11.
I20260807 03:39:25.388909 18064 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a59a83a64df475ea15dd572f84024bd with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 3 (current committed config index is 7)) from {username='slave'} at 127.0.0.1:33238
I20260807 03:39:25.390605 17898 catalog_manager.cc:4841] TS d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): tablet 9a59a83a64df475ea15dd572f84024bd (table test-workload [id=677ca051b50345e183d60556fc42291d]) successfully deleted
I20260807 03:39:25.416563 17914 catalog_manager.cc:5495] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 8 to 9, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 9 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:25.418942 17914 catalog_manager.cc:5495] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:39:25.440366 17915 catalog_manager.cc:5495] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 9 to 11, leader changed from d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 11 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:25.553486 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:39:25.603441 21277 raft_consensus.cc:1050] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49: attempting to promote NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 to VOTER
I20260807 03:39:25.606456 21277 consensus_queue.cc:237] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 1, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:25.633958 18183 raft_consensus.cc:1261] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260807 03:39:25.635253 21260 consensus_queue.cc:1035] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:39:25.647094 18080 raft_consensus.cc:1261] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260807 03:39:25.648936 19292 raft_consensus.cc:1261] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 3 LEARNER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260807 03:39:25.652539 21461 consensus_queue.cc:1035] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:39:25.653432 21260 consensus_queue.cc:1035] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:39:25.676610 18311 raft_consensus.cc:1261] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 11 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260807 03:39:25.689819 18295 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88"
dest_uuid: "66a36e1ac8e945e893668e282d394c49"
mode: GRACEFUL
new_leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:58736
I20260807 03:39:25.691973 21611 consensus_queue.cc:1035] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.002s
I20260807 03:39:25.694057 18295 raft_consensus.cc:595] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Received request to transfer leadership to TS d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:25.704222 18293 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616"
dest_uuid: "66a36e1ac8e945e893668e282d394c49"
mode: GRACEFUL
new_leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:58736
I20260807 03:39:25.704929 18293 raft_consensus.cc:595] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Received request to transfer leadership to TS d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:25.726827 18293 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4"
dest_uuid: "66a36e1ac8e945e893668e282d394c49"
mode: GRACEFUL
new_leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:58736
I20260807 03:39:25.727687 18293 raft_consensus.cc:595] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Received request to transfer leadership to TS d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:25.745925 18073 raft_consensus.cc:1261] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 11 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260807 03:39:25.772562 18293 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585"
dest_uuid: "66a36e1ac8e945e893668e282d394c49"
mode: GRACEFUL
new_leader_uuid: "3c4bceb836db4536a0f255fdf86abc02"
from {username='slave'} at 127.0.0.1:58736
I20260807 03:39:25.773232 18293 raft_consensus.cc:595] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Received request to transfer leadership to TS 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:25.792639 18295 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bb66518c65e4495e881ce860253d0801"
dest_uuid: "66a36e1ac8e945e893668e282d394c49"
mode: GRACEFUL
new_leader_uuid: "3c4bceb836db4536a0f255fdf86abc02"
from {username='slave'} at 127.0.0.1:58736
I20260807 03:39:25.793678 18295 raft_consensus.cc:595] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Received request to transfer leadership to TS 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:25.810071 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260807 03:39:25.812412 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:39:25.813261 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:39:25.776427 21611 consensus_queue.cc:1035] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.028s
W20260807 03:39:25.891997 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 100266us
I20260807 03:39:25.904937 21665 raft_consensus.cc:979] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49: : Instructing follower 3c4bceb836db4536a0f255fdf86abc02 to start an election
I20260807 03:39:25.906042 21461 raft_consensus.cc:1067] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Signalling peer 3c4bceb836db4536a0f255fdf86abc02 to start an election
I20260807 03:39:25.911890 18191 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bb66518c65e4495e881ce860253d0801"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
from {username='slave'} at 127.0.0.1:44660
I20260807 03:39:25.912472 18191 raft_consensus.cc:482] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:39:25.912799 18191 raft_consensus.cc:3037] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:25.921423 18191 raft_consensus.cc:504] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:25.925168 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:25.925768 18082 raft_consensus.cc:3037] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:25.927246 18191 leader_election.cc:288] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:25.940161 21527 raft_consensus.cc:2932] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:25.947726 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:25.948446 18311 raft_consensus.cc:3032] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:39:25.948748 18311 raft_consensus.cc:729] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:25.952343 18082 raft_consensus.cc:2454] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:39:25.952996 18070 raft_consensus.cc:1226] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:25.953027 18311 consensus_queue.cc:260] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:25.954607 18311 raft_consensus.cc:3037] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:25.956115 18140 leader_election.cc:302] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:25.957041 21610 raft_consensus.cc:2781] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 03:39:25.957577 21610 raft_consensus.cc:686] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:25.958421 21610 consensus_queue.cc:237] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:25.964036 18191 raft_consensus.cc:2932] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:25.977737 19292 raft_consensus.cc:2932] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:25.983353 18311 raft_consensus.cc:2454] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:39:26.009390 17912 catalog_manager.cc:5495] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 reported cstate change: config changed from index 3 to 4, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:26.018450 17914 catalog_manager.cc:5495] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 3 to 4, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 4 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:26.057278 18073 raft_consensus.cc:2932] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:26.065541 21166 raft_consensus.cc:979] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49: : Instructing follower d9716e2bf06442f5b3b0e81087c53c4d to start an election
I20260807 03:39:26.066083 21166 raft_consensus.cc:1067] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Signalling peer d9716e2bf06442f5b3b0e81087c53c4d to start an election
I20260807 03:39:26.071987 21664 raft_consensus.cc:979] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49: : Instructing follower 3c4bceb836db4536a0f255fdf86abc02 to start an election
I20260807 03:39:26.072510 21263 raft_consensus.cc:1067] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Signalling peer 3c4bceb836db4536a0f255fdf86abc02 to start an election
I20260807 03:39:26.076422 18070 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4"
dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:33254
I20260807 03:39:26.077009 18070 raft_consensus.cc:482] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:39:26.077354 18070 raft_consensus.cc:3037] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:26.078940 18191 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
from {username='slave'} at 127.0.0.1:44660
I20260807 03:39:26.079506 18191 raft_consensus.cc:482] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:39:26.079905 18191 raft_consensus.cc:3037] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:26.091799 18191 raft_consensus.cc:504] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:26.097390 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:26.097921 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:26.098114 18311 raft_consensus.cc:3032] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:39:26.098584 18080 raft_consensus.cc:3037] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:26.098627 18311 raft_consensus.cc:729] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:26.099890 18311 consensus_queue.cc:260] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:26.101131 18311 raft_consensus.cc:3037] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:26.111340 18080 raft_consensus.cc:2454] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:39:26.111946 18191 leader_election.cc:288] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:26.113551 18140 leader_election.cc:302] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:26.125794 21610 raft_consensus.cc:2781] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 03:39:26.126299 21610 raft_consensus.cc:686] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:26.127178 21610 consensus_queue.cc:237] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:26.130745 18311 raft_consensus.cc:2454] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:39:26.118685 18070 raft_consensus.cc:504] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:26.149675 18070 leader_election.cc:288] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:26.150660 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:26.180363 18293 raft_consensus.cc:3032] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:39:26.180689 18293 raft_consensus.cc:729] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:26.151314 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:26.165146 21263 raft_consensus.cc:979] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49: : Instructing follower d9716e2bf06442f5b3b0e81087c53c4d to start an election
I20260807 03:39:26.181466 18293 consensus_queue.cc:260] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:26.182638 21263 raft_consensus.cc:1067] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Signalling peer d9716e2bf06442f5b3b0e81087c53c4d to start an election
I20260807 03:39:26.183102 18293 raft_consensus.cc:3037] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:26.181499 18183 raft_consensus.cc:3037] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:26.186177 18070 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88"
dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:33254
I20260807 03:39:26.186694 18070 raft_consensus.cc:482] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:39:26.186718 21662 raft_consensus.cc:979] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49: : Instructing follower d9716e2bf06442f5b3b0e81087c53c4d to start an election
I20260807 03:39:26.187179 18070 raft_consensus.cc:3037] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:26.187391 21277 raft_consensus.cc:1067] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Signalling peer d9716e2bf06442f5b3b0e81087c53c4d to start an election
I20260807 03:39:26.189179 18073 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616"
dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:33254
I20260807 03:39:26.189678 18073 raft_consensus.cc:482] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:39:26.190075 18073 raft_consensus.cc:3037] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:26.190611 18183 raft_consensus.cc:2454] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 4.
I20260807 03:39:26.191785 18027 leader_election.cc:302] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:26.192649 21224 raft_consensus.cc:2781] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader election won for term 4
I20260807 03:39:26.193146 21224 raft_consensus.cc:686] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:26.190076 18293 raft_consensus.cc:2454] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 4.
I20260807 03:39:26.194090 21224 consensus_queue.cc:237] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:26.212205 18293 consensus_queue.cc:237] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:26.217137 18073 raft_consensus.cc:504] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:26.215618 18070 raft_consensus.cc:504] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:26.220995 18070 leader_election.cc:288] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:26.222019 18073 leader_election.cc:288] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:26.223537 19292 raft_consensus.cc:1261] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 3 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260807 03:39:26.225975 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: true dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:26.226521 18191 raft_consensus.cc:3037] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:26.227156 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: true dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:26.227707 18300 raft_consensus.cc:3032] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Stepping down as leader of term 7
I20260807 03:39:26.227993 18300 raft_consensus.cc:729] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:26.228701 18300 consensus_queue.cc:260] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:26.229816 18300 raft_consensus.cc:3037] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:26.229719 17914 catalog_manager.cc:5495] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 3 to 4, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 4 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:26.232566 21263 consensus_queue.cc:1035] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 03:39:26.235499 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:26.235741 18189 raft_consensus.cc:1261] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260807 03:39:26.236068 18295 raft_consensus.cc:3032] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:39:26.236421 18295 raft_consensus.cc:729] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:26.237087 18295 consensus_queue.cc:260] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:26.238245 18295 raft_consensus.cc:3037] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:26.232784 18191 raft_consensus.cc:2454] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 8.
I20260807 03:39:26.240985 18027 leader_election.cc:302] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:26.242501 21166 consensus_queue.cc:1035] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 03:39:26.244572 21363 raft_consensus.cc:2781] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader election won for term 8
I20260807 03:39:26.245199 21363 raft_consensus.cc:686] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:26.242861 17920 catalog_manager.cc:5495] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 3 to 4, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 4 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:26.248967 21363 consensus_queue.cc:237] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:26.264590 21277 raft_consensus.cc:2932] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:26.283124 18295 raft_consensus.cc:2454] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 4.
I20260807 03:39:26.293334 17899 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 3433ce141bc54c5395e00fe40cf9bb69 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 03:39:26.290530 19292 raft_consensus.cc:2932] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:26.287752 18300 raft_consensus.cc:2454] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 8.
I20260807 03:39:26.247740 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:26.295495 18183 raft_consensus.cc:3037] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:26.290478 18180 raft_consensus.cc:2932] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:26.294034 18027 leader_election.cc:302] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:26.300576 21581 raft_consensus.cc:2781] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader election won for term 4
I20260807 03:39:26.303220 21581 raft_consensus.cc:686] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:26.304154 21581 consensus_queue.cc:237] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:26.320576 17914 catalog_manager.cc:5495] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 7 to 8, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 8 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:26.349018 17920 catalog_manager.cc:5495] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: config changed from index 4 to 5, VOTER d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) evicted. New cstate: current_term: 3 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:26.359968 18064 tablet_service.cc:1514] Processing DeleteTablet for tablet 3433ce141bc54c5395e00fe40cf9bb69 with delete_type TABLET_DATA_TOMBSTONED (TS d9716e2bf06442f5b3b0e81087c53c4d not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33238
I20260807 03:39:26.375597 21689 tablet_replica.cc:330] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d: stopping tablet replica
I20260807 03:39:26.379177 17915 catalog_manager.cc:5495] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 3 to 4, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 4 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:26.397817 18183 raft_consensus.cc:2454] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 4.
I20260807 03:39:26.416467 21689 raft_consensus.cc:2229] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Raft consensus shutting down.
I20260807 03:39:26.419097 21689 raft_consensus.cc:2258] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 03:39:26.431398 21689 ts_tablet_manager.cc:1905] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 03:39:26.452646 21689 ts_tablet_manager.cc:1918] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.4
I20260807 03:39:26.453073 21689 log.cc:1196] T 3433ce141bc54c5395e00fe40cf9bb69 P d9716e2bf06442f5b3b0e81087c53c4d: Deleting WAL directory at /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786073845599630-16154-0/minicluster-data/ts-0-root/wals/3433ce141bc54c5395e00fe40cf9bb69
I20260807 03:39:26.454627 17898 catalog_manager.cc:4841] TS d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): tablet 3433ce141bc54c5395e00fe40cf9bb69 (table test-workload [id=677ca051b50345e183d60556fc42291d]) successfully deleted
I20260807 03:39:26.515583 18073 raft_consensus.cc:1261] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260807 03:39:26.517123 21611 consensus_queue.cc:1035] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:39:26.542896 18295 raft_consensus.cc:1261] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260807 03:39:26.544543 21611 consensus_queue.cc:1035] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:39:26.878863 18180 raft_consensus.cc:1261] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260807 03:39:26.880725 21581 consensus_queue.cc:1035] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 03:39:26.901288 18193 raft_consensus.cc:1261] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260807 03:39:26.909130 21224 consensus_queue.cc:1035] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:39:26.947516 18180 raft_consensus.cc:1261] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 03:39:26.949227 21714 consensus_queue.cc:1035] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 03:39:26.963516 18073 raft_consensus.cc:1261] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260807 03:39:26.968549 21610 consensus_queue.cc:1035] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 03:39:26.981721 18297 raft_consensus.cc:1261] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260807 03:39:26.988734 18300 raft_consensus.cc:1261] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260807 03:39:27.002568 18300 raft_consensus.cc:1261] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 03:39:27.004546 18295 raft_consensus.cc:1261] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260807 03:39:27.010993 21610 consensus_queue.cc:1035] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:39:27.032454 21518 consensus_queue.cc:1035] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260807 03:39:27.044270 21724 log.cc:924] Time spent T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.083s user 0.001s sys 0.000s
I20260807 03:39:27.058321 21714 consensus_queue.cc:1035] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:39:27.068799 21714 consensus_queue.cc:1035] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 03:39:27.131102 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:39:27.237000 18300 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79"
dest_uuid: "66a36e1ac8e945e893668e282d394c49"
mode: GRACEFUL
new_leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:58736
I20260807 03:39:27.237561 18300 raft_consensus.cc:595] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Received request to transfer leadership to TS d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:27.240177 18300 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917"
dest_uuid: "66a36e1ac8e945e893668e282d394c49"
mode: GRACEFUL
new_leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:58736
I20260807 03:39:27.240731 18300 raft_consensus.cc:595] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Received request to transfer leadership to TS d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:27.242962 18300 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "60437eed8852484c88e84993e376670e"
dest_uuid: "66a36e1ac8e945e893668e282d394c49"
mode: GRACEFUL
new_leader_uuid: "3c4bceb836db4536a0f255fdf86abc02"
from {username='slave'} at 127.0.0.1:58736
I20260807 03:39:27.243557 18300 raft_consensus.cc:595] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Received request to transfer leadership to TS 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:27.245869 18300 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a"
dest_uuid: "66a36e1ac8e945e893668e282d394c49"
mode: GRACEFUL
new_leader_uuid: "3c4bceb836db4536a0f255fdf86abc02"
from {username='slave'} at 127.0.0.1:58736
I20260807 03:39:27.246464 18300 raft_consensus.cc:595] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Received request to transfer leadership to TS 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:27.272347 18082 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d0047935053942d480774d46b081709d"
dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
mode: GRACEFUL
new_leader_uuid: "3c4bceb836db4536a0f255fdf86abc02"
from {username='slave'} at 127.0.0.1:55742
I20260807 03:39:27.273001 18082 raft_consensus.cc:595] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 8 LEADER]: Received request to transfer leadership to TS 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:27.276119 18300 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7"
dest_uuid: "66a36e1ac8e945e893668e282d394c49"
mode: GRACEFUL
new_leader_uuid: "3c4bceb836db4536a0f255fdf86abc02"
from {username='slave'} at 127.0.0.1:58736
I20260807 03:39:27.276775 18300 raft_consensus.cc:595] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Received request to transfer leadership to TS 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:27.280435 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260807 03:39:27.282433 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:39:27.283087 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:39:27.430531 21662 raft_consensus.cc:979] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49: : Instructing follower d9716e2bf06442f5b3b0e81087c53c4d to start an election
I20260807 03:39:27.431072 21707 raft_consensus.cc:1067] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Signalling peer d9716e2bf06442f5b3b0e81087c53c4d to start an election
I20260807 03:39:27.435245 18082 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79"
dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:33254
I20260807 03:39:27.435755 18082 raft_consensus.cc:482] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:39:27.436107 18082 raft_consensus.cc:3037] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:27.442768 18082 raft_consensus.cc:504] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:27.447930 18082 leader_election.cc:288] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:27.461941 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:27.462531 18300 raft_consensus.cc:3032] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 03:39:27.462790 18300 raft_consensus.cc:729] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:27.462956 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:27.463903 18183 raft_consensus.cc:3037] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:27.463577 18300 consensus_queue.cc:260] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:27.470675 18300 raft_consensus.cc:3037] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:27.481307 21714 raft_consensus.cc:979] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d: : Instructing follower 3c4bceb836db4536a0f255fdf86abc02 to start an election
I20260807 03:39:27.481918 21714 raft_consensus.cc:1067] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 8 LEADER]: Signalling peer 3c4bceb836db4536a0f255fdf86abc02 to start an election
I20260807 03:39:27.493005 18300 raft_consensus.cc:2454] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 5.
I20260807 03:39:27.493958 18180 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d0047935053942d480774d46b081709d"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
from {username='slave'} at 127.0.0.1:44662
I20260807 03:39:27.494608 18180 raft_consensus.cc:482] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:39:27.494977 18180 raft_consensus.cc:3037] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:27.497387 18183 raft_consensus.cc:2454] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 5.
I20260807 03:39:27.500330 18027 leader_election.cc:302] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:27.529917 21714 raft_consensus.cc:2781] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election won for term 5
I20260807 03:39:27.530517 21714 raft_consensus.cc:686] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:27.531394 21714 consensus_queue.cc:237] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:27.568905 17915 catalog_manager.cc:5495] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 4 to 5, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 5 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:27.598822 18180 raft_consensus.cc:504] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:27.622433 21661 raft_consensus.cc:979] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49: : Instructing follower d9716e2bf06442f5b3b0e81087c53c4d to start an election
I20260807 03:39:27.622937 21661 raft_consensus.cc:1067] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Signalling peer d9716e2bf06442f5b3b0e81087c53c4d to start an election
I20260807 03:39:27.626431 18073 consensus_queue.cc:237] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:39:27.632869 18180 leader_election.cc:288] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:27.635977 18082 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917"
dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:33254
I20260807 03:39:27.636606 18082 raft_consensus.cc:482] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:39:27.637058 18082 raft_consensus.cc:3037] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 6
W20260807 03:39:27.645511 18025 consensus_peers.cc:487] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d -> Peer 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381): Couldn't send request to peer 5c9b5062bdfa46af9396f1a85025a331. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d24d0bbea8bd49d482ad95538ca6e631. This is attempt 1: this message will repeat every 5th retry.
I20260807 03:39:27.651141 21662 raft_consensus.cc:979] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49: : Instructing follower 3c4bceb836db4536a0f255fdf86abc02 to start an election
I20260807 03:39:27.651904 21662 raft_consensus.cc:1067] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Signalling peer 3c4bceb836db4536a0f255fdf86abc02 to start an election
I20260807 03:39:27.654501 18193 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "60437eed8852484c88e84993e376670e"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
from {username='slave'} at 127.0.0.1:44660
I20260807 03:39:27.655175 18193 raft_consensus.cc:482] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:39:27.655529 18193 raft_consensus.cc:3037] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:27.672956 18183 raft_consensus.cc:1261] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260807 03:39:27.674386 21714 consensus_queue.cc:1035] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:39:27.684659 18193 raft_consensus.cc:504] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:27.686901 18193 leader_election.cc:288] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:27.694509 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: true dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:27.695060 18073 raft_consensus.cc:3032] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 8 LEADER]: Stepping down as leader of term 8
I20260807 03:39:27.695353 18073 raft_consensus.cc:729] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:27.697340 18297 raft_consensus.cc:1261] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260807 03:39:27.700145 18073 consensus_queue.cc:260] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:27.701871 18073 raft_consensus.cc:3037] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:27.706779 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: true dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:27.707540 18311 raft_consensus.cc:3037] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:27.710078 18073 raft_consensus.cc:2454] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 9.
I20260807 03:39:27.714016 18311 raft_consensus.cc:2454] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 9.
I20260807 03:39:27.715724 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:27.716596 18297 raft_consensus.cc:3032] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 03:39:27.716910 18297 raft_consensus.cc:729] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:27.717744 18297 consensus_queue.cc:260] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:27.719323 18297 raft_consensus.cc:3037] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:27.720196 21714 consensus_queue.cc:1035] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:39:27.721267 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:27.721803 18070 raft_consensus.cc:3037] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:27.724735 18141 leader_election.cc:302] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:27.725594 21700 raft_consensus.cc:2781] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader election won for term 9
I20260807 03:39:27.726078 21700 raft_consensus.cc:686] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 9 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:27.725449 21703 raft_consensus.cc:2932] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 LEADER]: Committing config change with OpId 4.4: config changed from index -1 to 4, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:39:27.726871 21700 consensus_queue.cc:237] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:27.728073 18297 raft_consensus.cc:2454] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 6.
I20260807 03:39:27.730218 18189 raft_consensus.cc:2932] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Committing config change with OpId 4.4: config changed from index -1 to 4, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:39:27.735847 18141 leader_election.cc:302] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:27.736593 21610 raft_consensus.cc:2781] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 03:39:27.737035 21610 raft_consensus.cc:686] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:27.737915 21716 raft_consensus.cc:979] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49: : Instructing follower 3c4bceb836db4536a0f255fdf86abc02 to start an election
I20260807 03:39:27.737823 21610 consensus_queue.cc:237] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:27.738400 21716 raft_consensus.cc:1067] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Signalling peer 3c4bceb836db4536a0f255fdf86abc02 to start an election
I20260807 03:39:27.745687 18070 raft_consensus.cc:2454] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 6.
I20260807 03:39:27.747200 17915 catalog_manager.cc:5495] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: config changed from index -1 to 4, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New cstate: current_term: 4 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:27.750797 18082 raft_consensus.cc:504] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:27.751168 18180 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
from {username='slave'} at 127.0.0.1:44660
I20260807 03:39:27.764313 18180 raft_consensus.cc:482] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:39:27.764652 18180 raft_consensus.cc:3037] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:27.768333 18073 consensus_queue.cc:237] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:39:27.770110 18082 leader_election.cc:288] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:27.772169 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:27.781766 18311 raft_consensus.cc:3032] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 03:39:27.782094 18311 raft_consensus.cc:729] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:27.782816 18311 consensus_queue.cc:260] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:27.784030 18311 raft_consensus.cc:3037] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:27.786691 18180 raft_consensus.cc:504] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:27.794348 18311 raft_consensus.cc:2454] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 6.
I20260807 03:39:27.796962 18180 leader_election.cc:288] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:27.802497 21661 raft_consensus.cc:979] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49: : Instructing follower 3c4bceb836db4536a0f255fdf86abc02 to start an election
I20260807 03:39:27.802991 21661 raft_consensus.cc:1067] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Signalling peer 3c4bceb836db4536a0f255fdf86abc02 to start an election
I20260807 03:39:27.807960 18180 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
from {username='slave'} at 127.0.0.1:44660
I20260807 03:39:27.808614 18180 raft_consensus.cc:482] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:39:27.808997 18180 raft_consensus.cc:3037] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:27.809484 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:27.810081 18189 raft_consensus.cc:3037] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:27.807974 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:27.812129 18311 raft_consensus.cc:3032] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Stepping down as leader of term 7
I20260807 03:39:27.812424 18311 raft_consensus.cc:729] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:27.813192 18311 consensus_queue.cc:260] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:27.818207 18189 raft_consensus.cc:2454] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 6.
I20260807 03:39:27.844513 18311 raft_consensus.cc:3037] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:27.834926 18180 raft_consensus.cc:504] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:27.902602 18180 leader_election.cc:288] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:27.909909 19324 debug-util.cc:398] Leaking SignalData structure 0x7b08002e2060 after lost signal to thread 16157
W20260807 03:39:27.911998 19324 debug-util.cc:398] Leaking SignalData structure 0x7b08002e2740 after lost signal to thread 17976
I20260807 03:39:27.916774 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:27.918393 18297 raft_consensus.cc:3032] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 03:39:27.918889 18297 raft_consensus.cc:729] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:27.921701 18297 consensus_queue.cc:260] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:27.922552 19324 debug-util.cc:398] Leaking SignalData structure 0x7b08002f3660 after lost signal to thread 18113
I20260807 03:39:27.924594 18297 raft_consensus.cc:3037] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
W20260807 03:39:27.929227 19324 debug-util.cc:398] Leaking SignalData structure 0x7b08003437a0 after lost signal to thread 18228
W20260807 03:39:27.954586 18025 consensus_peers.cc:487] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d -> Peer 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381): Couldn't send request to peer 5c9b5062bdfa46af9396f1a85025a331. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6129789267024fac8a337070c10a472a. This is attempt 1: this message will repeat every 5th retry.
W20260807 03:39:27.932078 19324 debug-util.cc:398] Leaking SignalData structure 0x7b08000c8280 after lost signal to thread 18346
I20260807 03:39:27.954070 17914 catalog_manager.cc:5495] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 8 to 9, leader changed from d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 9 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:27.964216 17914 catalog_manager.cc:5495] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 5 to 6, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 6 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
W20260807 03:39:27.981551 19324 debug-util.cc:398] Leaking SignalData structure 0x7b080036cd80 after lost signal to thread 19330
I20260807 03:39:27.989847 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:27.990808 18071 raft_consensus.cc:3037] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:27.998849 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:28.006937 18080 raft_consensus.cc:3037] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:27.998822 18027 leader_election.cc:302] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:28.014052 18311 raft_consensus.cc:2454] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 8.
I20260807 03:39:28.023649 18080 raft_consensus.cc:2454] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 6.
I20260807 03:39:28.023792 18071 raft_consensus.cc:2454] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 8.
I20260807 03:39:28.024638 18141 leader_election.cc:302] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:28.032631 18297 raft_consensus.cc:2454] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 6.
I20260807 03:39:28.034197 18295 raft_consensus.cc:1261] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260807 03:39:28.045869 21763 raft_consensus.cc:2781] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader election won for term 6
I20260807 03:39:28.047858 21546 raft_consensus.cc:2781] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader election won for term 8
I20260807 03:39:28.050580 21546 raft_consensus.cc:686] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:28.051983 21546 consensus_queue.cc:237] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:28.060719 18191 raft_consensus.cc:1261] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 03:39:28.066212 18193 raft_consensus.cc:1261] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260807 03:39:28.073660 21763 raft_consensus.cc:686] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:28.074951 21763 consensus_queue.cc:237] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:28.123919 21518 consensus_queue.cc:1035] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 03:39:28.136981 18140 leader_election.cc:302] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:28.141956 21771 raft_consensus.cc:2781] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 03:39:28.143520 21771 raft_consensus.cc:686] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:28.158602 21771 consensus_queue.cc:237] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:28.156559 21768 ts_tablet_manager.cc:927] T d24d0bbea8bd49d482ad95538ca6e631 P 5c9b5062bdfa46af9396f1a85025a331: Initiating tablet copy from peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163)
W20260807 03:39:28.156770 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 53890.5us
I20260807 03:39:28.274613 18071 raft_consensus.cc:1261] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260807 03:39:28.272233 17915 catalog_manager.cc:5495] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 5 to 6, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 6 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
W20260807 03:39:28.296579 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 63563.1us
I20260807 03:39:28.298674 21714 consensus_queue.cc:1035] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20260807 03:39:28.302181 21769 consensus_queue.cc:1035] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 03:39:28.303248 21714 consensus_queue.cc:1035] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 03:39:28.306986 18295 raft_consensus.cc:1261] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260807 03:39:28.316031 18295 raft_consensus.cc:2932] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Committing config change with OpId 4.4: config changed from index -1 to 4, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:39:28.318008 21768 tablet_copy_client.cc:316] T d24d0bbea8bd49d482ad95538ca6e631 P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Beginning tablet copy session from remote peer at address 127.15.198.129:36163
I20260807 03:39:28.371529 18094 tablet_copy_service.cc:140] P d9716e2bf06442f5b3b0e81087c53c4d: Received BeginTabletCopySession request for tablet d24d0bbea8bd49d482ad95538ca6e631 from peer 5c9b5062bdfa46af9396f1a85025a331 ({username='slave'} at 127.0.0.1:58950)
I20260807 03:39:28.372388 18094 tablet_copy_service.cc:161] P d9716e2bf06442f5b3b0e81087c53c4d: Beginning new tablet copy session on tablet d24d0bbea8bd49d482ad95538ca6e631 from peer 5c9b5062bdfa46af9396f1a85025a331 at {username='slave'} at 127.0.0.1:58950: session id = 5c9b5062bdfa46af9396f1a85025a331-d24d0bbea8bd49d482ad95538ca6e631
I20260807 03:39:28.380306 18312 raft_consensus.cc:1261] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 03:39:28.451093 18094 tablet_copy_source_session.cc:205] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 03:39:28.455277 21768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d24d0bbea8bd49d482ad95538ca6e631. 1 dirs total, 0 dirs full, 0 dirs failed
W20260807 03:39:28.456566 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 74279.2us
I20260807 03:39:28.461849 18082 raft_consensus.cc:1261] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 03:39:28.499090 17914 catalog_manager.cc:5495] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 7 to 8, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 8 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:28.505257 17914 catalog_manager.cc:5495] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 5 to 6, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 6 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:28.489044 21771 consensus_queue.cc:1035] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
I20260807 03:39:28.524339 21757 consensus_queue.cc:1035] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.011s
I20260807 03:39:28.526579 21786 ts_tablet_manager.cc:927] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331: Initiating tablet copy from peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163)
I20260807 03:39:28.547459 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:39:28.721055 18295 raft_consensus.cc:1261] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 03:39:28.731159 21786 tablet_copy_client.cc:316] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Beginning tablet copy session from remote peer at address 127.15.198.129:36163
I20260807 03:39:28.801733 21760 consensus_queue.cc:1035] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.081s
W20260807 03:39:28.873548 21788 log.cc:924] Time spent T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20260807 03:39:28.873862 18094 tablet_copy_service.cc:140] P d9716e2bf06442f5b3b0e81087c53c4d: Received BeginTabletCopySession request for tablet 6129789267024fac8a337070c10a472a from peer 5c9b5062bdfa46af9396f1a85025a331 ({username='slave'} at 127.0.0.1:58950)
I20260807 03:39:28.874531 18094 tablet_copy_service.cc:161] P d9716e2bf06442f5b3b0e81087c53c4d: Beginning new tablet copy session on tablet 6129789267024fac8a337070c10a472a from peer 5c9b5062bdfa46af9396f1a85025a331 at {username='slave'} at 127.0.0.1:58950: session id = 5c9b5062bdfa46af9396f1a85025a331-6129789267024fac8a337070c10a472a
W20260807 03:39:28.887221 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 319949us
I20260807 03:39:28.895618 18094 tablet_copy_source_session.cc:205] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 03:39:28.904361 21768 tablet_copy_client.cc:791] T d24d0bbea8bd49d482ad95538ca6e631 P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Starting download of 0 data blocks...
I20260807 03:39:28.905220 21768 tablet_copy_client.cc:655] T d24d0bbea8bd49d482ad95538ca6e631 P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Starting download of 1 WAL segments...
I20260807 03:39:28.923900 21768 tablet_copy_client.cc:531] T d24d0bbea8bd49d482ad95538ca6e631 P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 03:39:28.935336 21786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6129789267024fac8a337070c10a472a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:39:28.955132 18080 raft_consensus.cc:1261] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260807 03:39:28.959105 18080 raft_consensus.cc:1261] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 03:39:28.974078 21791 consensus_queue.cc:1035] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 03:39:28.974931 18295 raft_consensus.cc:1261] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260807 03:39:28.976713 18311 raft_consensus.cc:1261] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260807 03:39:28.984355 21785 log.cc:924] Time spent T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.179s user 0.003s sys 0.020s
I20260807 03:39:29.003829 21771 consensus_queue.cc:1035] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 03:39:29.009503 21792 consensus_queue.cc:1035] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260807 03:39:29.020290 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 268703us
I20260807 03:39:29.054150 21760 raft_consensus.cc:2932] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 9 LEADER]: Committing config change with OpId 9.6: config changed from index -1 to 6, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:39:29.063087 18183 raft_consensus.cc:1261] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260807 03:39:29.133942 21714 consensus_queue.cc:1035] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 03:39:29.135671 21768 tablet_bootstrap.cc:492] T d24d0bbea8bd49d482ad95538ca6e631 P 5c9b5062bdfa46af9396f1a85025a331: Bootstrap starting.
I20260807 03:39:29.140599 21546 consensus_queue.cc:1035] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 03:39:29.143214 21769 consensus_queue.cc:1035] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:39:29.164645 18300 raft_consensus.cc:1261] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260807 03:39:29.165405 18298 raft_consensus.cc:2932] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Committing config change with OpId 9.6: config changed from index -1 to 6, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:39:29.175380 21786 tablet_copy_client.cc:791] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Starting download of 0 data blocks...
I20260807 03:39:29.238045 21786 tablet_copy_client.cc:655] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Starting download of 1 WAL segments...
I20260807 03:39:29.231439 18187 raft_consensus.cc:2932] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Committing config change with OpId 9.6: config changed from index -1 to 6, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:39:29.247342 21786 tablet_copy_client.cc:531] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 03:39:29.327453 18311 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d"
dest_uuid: "66a36e1ac8e945e893668e282d394c49"
mode: GRACEFUL
new_leader_uuid: "3c4bceb836db4536a0f255fdf86abc02"
from {username='slave'} at 127.0.0.1:58736
I20260807 03:39:29.328236 18311 raft_consensus.cc:595] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Received request to transfer leadership to TS 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:29.329216 21786 tablet_bootstrap.cc:492] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331: Bootstrap starting.
I20260807 03:39:29.331959 18187 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4ae0265e0707432980672f7cc955c571"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
mode: GRACEFUL
new_leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:34758
I20260807 03:39:29.332619 18187 raft_consensus.cc:595] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 LEADER]: Received request to transfer leadership to TS d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:29.353374 17914 catalog_manager.cc:5495] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 reported cstate change: config changed from index -1 to 6, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New cstate: current_term: 9 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
W20260807 03:39:29.399803 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 64170.9us
I20260807 03:39:29.343523 21763 consensus_queue.cc:1035] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.057s
I20260807 03:39:29.412004 18082 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1"
dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
mode: GRACEFUL
new_leader_uuid: "66a36e1ac8e945e893668e282d394c49"
from {username='slave'} at 127.0.0.1:55742
I20260807 03:39:29.432583 18082 raft_consensus.cc:595] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 LEADER]: Received request to transfer leadership to TS 66a36e1ac8e945e893668e282d394c49
W20260807 03:39:29.482535 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 73525.1us
I20260807 03:39:29.497856 21770 raft_consensus.cc:979] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02: : Instructing follower d9716e2bf06442f5b3b0e81087c53c4d to start an election
I20260807 03:39:29.498468 21770 raft_consensus.cc:1067] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 LEADER]: Signalling peer d9716e2bf06442f5b3b0e81087c53c4d to start an election
I20260807 03:39:29.500730 18082 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4ae0265e0707432980672f7cc955c571"
dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:33268
I20260807 03:39:29.501297 18082 raft_consensus.cc:482] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:39:29.509361 18082 raft_consensus.cc:3037] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 6
W20260807 03:39:29.525125 21837 log.cc:924] Time spent T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.069s user 0.001s sys 0.000s
W20260807 03:39:29.526321 21839 log.cc:924] Time spent T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.115s user 0.000s sys 0.000s
I20260807 03:39:29.536692 18311 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904"
dest_uuid: "66a36e1ac8e945e893668e282d394c49"
mode: GRACEFUL
new_leader_uuid: "3c4bceb836db4536a0f255fdf86abc02"
from {username='slave'} at 127.0.0.1:58736
I20260807 03:39:29.537303 18311 raft_consensus.cc:595] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Received request to transfer leadership to TS 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:29.529968 21760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:55748 (request call id 1) took 1769 ms. Trace:
I20260807 03:39:29.581640 18082 raft_consensus.cc:504] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:29.584386 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 46468us
I20260807 03:39:29.589556 18082 leader_election.cc:288] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:29.605809 21842 log.cc:924] Time spent T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
W20260807 03:39:29.612218 21838 log.cc:924] Time spent T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.135s user 0.000s sys 0.001s
I20260807 03:39:29.549494 21760 rpcz_store.cc:276] 0807 03:39:27.744938 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:27.757399 (+ 12461us) service_pool.cc:227] Handling call
0807 03:39:27.778326 (+ 20927us) log.cc:842] Serialized 513 byte log entry
0807 03:39:29.507138 (+1728812us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8782,"raft.queue_time_us":30238,"raft.run_cpu_time_us":2993,"raft.run_wall_time_us":24895,"thread_start_us":137150,"threads_started":2,"wal-append.queue_time_us":66890,"wal-append.run_cpu_time_us":7890,"wal-append.run_wall_time_us":1038190}
I20260807 03:39:29.643643 18193 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
mode: GRACEFUL
new_leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:34758
I20260807 03:39:29.646095 18193 raft_consensus.cc:595] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 LEADER]: Received request to transfer leadership to TS d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:29.652226 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260807 03:39:29.654233 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:39:29.655046 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:39:29.656309 21860 raft_consensus.cc:979] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02: : Instructing follower d9716e2bf06442f5b3b0e81087c53c4d to start an election
I20260807 03:39:29.656827 21860 raft_consensus.cc:1067] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 LEADER]: Signalling peer d9716e2bf06442f5b3b0e81087c53c4d to start an election
I20260807 03:39:29.659755 18069 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7"
dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:33268
I20260807 03:39:29.660308 18069 raft_consensus.cc:482] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:39:29.660624 18069 raft_consensus.cc:3037] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:29.691656 21277 raft_consensus.cc:979] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49: : Instructing follower 3c4bceb836db4536a0f255fdf86abc02 to start an election
I20260807 03:39:29.692224 21277 raft_consensus.cc:1067] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Signalling peer 3c4bceb836db4536a0f255fdf86abc02 to start an election
W20260807 03:39:29.713012 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 127830us
I20260807 03:39:29.712085 18069 raft_consensus.cc:504] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:29.736933 18069 leader_election.cc:288] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:29.744881 21786 tablet_bootstrap.cc:700] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331: Time spent opening tablet: real 0.133s user 0.004s sys 0.000s
W20260807 03:39:29.808949 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 81744.5us
I20260807 03:39:29.818848 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:29.819463 18193 raft_consensus.cc:3032] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 03:39:29.819787 18193 raft_consensus.cc:729] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:29.820506 18193 consensus_queue.cc:260] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:29.825850 18193 raft_consensus.cc:3037] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:29.844440 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:29.845047 18187 raft_consensus.cc:3032] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 LEADER]: Stepping down as leader of term 8
I20260807 03:39:29.845353 18187 raft_consensus.cc:729] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:29.846144 18193 raft_consensus.cc:2454] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 6.
I20260807 03:39:29.846045 18187 consensus_queue.cc:260] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:29.847056 18187 raft_consensus.cc:3037] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:29.852080 21768 tablet_bootstrap.cc:492] T d24d0bbea8bd49d482ad95538ca6e631 P 5c9b5062bdfa46af9396f1a85025a331: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 03:39:29.853082 21768 tablet_bootstrap.cc:492] T d24d0bbea8bd49d482ad95538ca6e631 P 5c9b5062bdfa46af9396f1a85025a331: Bootstrap complete.
I20260807 03:39:29.853520 18187 raft_consensus.cc:2454] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 9.
I20260807 03:39:29.853842 21768 ts_tablet_manager.cc:1397] T d24d0bbea8bd49d482ad95538ca6e631 P 5c9b5062bdfa46af9396f1a85025a331: Time spent bootstrapping tablet: real 0.718s user 0.025s sys 0.031s
W20260807 03:39:29.859076 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 60365.9us
I20260807 03:39:29.861336 21775 raft_consensus.cc:979] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49: : Instructing follower 3c4bceb836db4536a0f255fdf86abc02 to start an election
I20260807 03:39:29.867878 18183 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
from {username='slave'} at 127.0.0.1:44660
I20260807 03:39:29.868633 18183 raft_consensus.cc:482] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:39:29.869051 18183 raft_consensus.cc:3037] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:29.880240 21662 raft_consensus.cc:1067] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Signalling peer 3c4bceb836db4536a0f255fdf86abc02 to start an election
I20260807 03:39:29.857035 21768 raft_consensus.cc:348] T d24d0bbea8bd49d482ad95538ca6e631 P 5c9b5062bdfa46af9396f1a85025a331 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:39:29.894788 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:29.903023 21768 raft_consensus.cc:729] T d24d0bbea8bd49d482ad95538ca6e631 P 5c9b5062bdfa46af9396f1a85025a331 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 5c9b5062bdfa46af9396f1a85025a331, State: Initialized, Role: LEARNER
I20260807 03:39:29.903193 18312 raft_consensus.cc:3037] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:29.903772 21768 consensus_queue.cc:260] T d24d0bbea8bd49d482ad95538ca6e631 P 5c9b5062bdfa46af9396f1a85025a331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:39:29.902529 18180 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
from {username='slave'} at 127.0.0.1:44660
I20260807 03:39:29.906855 18180 raft_consensus.cc:482] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:39:29.907230 18180 raft_consensus.cc:3037] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:29.910043 21768 ts_tablet_manager.cc:1428] T d24d0bbea8bd49d482ad95538ca6e631 P 5c9b5062bdfa46af9396f1a85025a331: Time spent starting tablet: real 0.056s user 0.008s sys 0.000s
I20260807 03:39:29.940239 18312 raft_consensus.cc:2454] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 6.
I20260807 03:39:29.948302 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:29.948918 18295 raft_consensus.cc:3037] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:29.951076 18027 leader_election.cc:302] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:29.953876 18180 raft_consensus.cc:504] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:29.967608 18027 leader_election.cc:302] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:29.968700 18180 leader_election.cc:288] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:29.965170 18183 raft_consensus.cc:504] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:29.968627 21880 raft_consensus.cc:2781] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader election won for term 9
I20260807 03:39:29.981243 18183 leader_election.cc:288] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:29.990528 21824 raft_consensus.cc:2781] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader election won for term 6
I20260807 03:39:30.000481 18094 tablet_copy_service.cc:342] P d9716e2bf06442f5b3b0e81087c53c4d: Request end of tablet copy session 5c9b5062bdfa46af9396f1a85025a331-d24d0bbea8bd49d482ad95538ca6e631 received from {username='slave'} at 127.0.0.1:58950
I20260807 03:39:30.001111 18094 tablet_copy_service.cc:433] P d9716e2bf06442f5b3b0e81087c53c4d: ending tablet copy session 5c9b5062bdfa46af9396f1a85025a331-d24d0bbea8bd49d482ad95538ca6e631 on tablet d24d0bbea8bd49d482ad95538ca6e631 with peer 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:39:30.000560 21816 raft_consensus.cc:482] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:30.011502 18180 raft_consensus.cc:1226] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 4. Current term is 5. Ops: []
I20260807 03:39:30.012437 18312 raft_consensus.cc:1226] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:30.010978 21816 raft_consensus.cc:504] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:29.989976 18298 consensus_queue.cc:237] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:39:29.997413 21880 raft_consensus.cc:686] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:30.022075 21880 consensus_queue.cc:237] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:30.024298 21861 consensus_queue.cc:1046] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 03:39:30.006114 21824 raft_consensus.cc:686] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:30.020661 18295 raft_consensus.cc:2454] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 9.
I20260807 03:39:30.030905 18312 raft_consensus.cc:1226] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 8. Current term is 9. Ops: []
I20260807 03:39:30.032473 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:30.033041 18305 raft_consensus.cc:3032] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 03:39:30.033313 18305 raft_consensus.cc:729] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:30.033334 21824 consensus_queue.cc:237] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:30.034050 18305 consensus_queue.cc:260] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:30.035328 18305 raft_consensus.cc:3037] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:30.047122 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:30.047736 18310 raft_consensus.cc:3032] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 03:39:30.048026 18310 raft_consensus.cc:729] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:30.049008 18310 consensus_queue.cc:260] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:30.050608 18310 raft_consensus.cc:3037] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:30.051517 18305 raft_consensus.cc:2454] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 5.
I20260807 03:39:30.059554 21884 raft_consensus.cc:979] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d: : Instructing follower 66a36e1ac8e945e893668e282d394c49 to start an election
I20260807 03:39:30.074070 18141 leader_election.cc:302] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:30.078727 21518 raft_consensus.cc:1067] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 LEADER]: Signalling peer 66a36e1ac8e945e893668e282d394c49 to start an election
I20260807 03:39:30.082472 21816 leader_election.cc:288] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:30.084543 21796 raft_consensus.cc:2781] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election won for term 5
W20260807 03:39:30.077864 21805 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20260807 03:39:30.098013 18310 raft_consensus.cc:2454] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 6.
I20260807 03:39:30.118065 21796 raft_consensus.cc:686] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 5 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:30.119371 21796 consensus_queue.cc:237] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:30.133260 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 51537.5us
I20260807 03:39:30.134599 18141 leader_election.cc:302] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:30.136224 21864 raft_consensus.cc:2781] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 03:39:30.136703 21864 raft_consensus.cc:686] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:30.137826 21864 consensus_queue.cc:237] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:30.140435 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:30.098840 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:30.142311 18079 raft_consensus.cc:3037] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:30.099050 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:30.146440 18080 raft_consensus.cc:3037] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:30.112736 18312 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1"
dest_uuid: "66a36e1ac8e945e893668e282d394c49"
from {username='slave'} at 127.0.0.1:57276
I20260807 03:39:30.152446 18312 raft_consensus.cc:482] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:39:30.152905 18312 raft_consensus.cc:3037] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:30.113631 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:30.162166 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 47016.9us
I20260807 03:39:30.174100 18249 leader_election.cc:302] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:30.177054 21816 raft_consensus.cc:2726] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:30.193197 17918 catalog_manager.cc:5495] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 8 to 9, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 9 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:30.221623 21786 tablet_bootstrap.cc:492] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=1 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260807 03:39:30.222736 21786 tablet_bootstrap.cc:492] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331: Bootstrap complete.
I20260807 03:39:30.223539 21786 ts_tablet_manager.cc:1397] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331: Time spent bootstrapping tablet: real 0.895s user 0.041s sys 0.025s
I20260807 03:39:30.234082 18080 raft_consensus.cc:2454] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 5.
I20260807 03:39:30.239662 18079 raft_consensus.cc:2454] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 6.
I20260807 03:39:30.241194 18312 raft_consensus.cc:504] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:30.247608 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:30.249572 18079 raft_consensus.cc:3032] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 LEADER]: Stepping down as leader of term 2
I20260807 03:39:30.250211 18079 raft_consensus.cc:729] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:30.252406 18312 leader_election.cc:288] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:30.258030 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:30.258514 18180 raft_consensus.cc:1261] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260807 03:39:30.259855 18187 raft_consensus.cc:3037] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:30.268332 18079 consensus_queue.cc:260] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:30.290541 19297 raft_consensus.cc:1203] T d24d0bbea8bd49d482ad95538ca6e631 P 5c9b5062bdfa46af9396f1a85025a331 [term 4 LEARNER]: Deduplicated request from leader. Original: 4.3->[4.4-4.4] Dedup: 4.4->[]
I20260807 03:39:30.291494 18079 raft_consensus.cc:3037] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:30.291042 21786 raft_consensus.cc:348] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [term 9 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:30.295173 21786 raft_consensus.cc:729] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [term 9 LEARNER]: Becoming Follower/Learner. State: Replica: 5c9b5062bdfa46af9396f1a85025a331, State: Initialized, Role: LEARNER
I20260807 03:39:30.295821 21786 consensus_queue.cc:260] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:39:30.293720 18187 raft_consensus.cc:2454] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:39:30.311992 18079 raft_consensus.cc:2454] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:39:30.324043 18248 leader_election.cc:302] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
W20260807 03:39:30.410472 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 150720us
W20260807 03:39:30.423285 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 99361us
I20260807 03:39:30.422411 17914 catalog_manager.cc:5495] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 4 to 5, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 5 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:30.432868 18079 raft_consensus.cc:1261] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260807 03:39:30.435714 21873 raft_consensus.cc:2781] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:39:30.466085 21786 ts_tablet_manager.cc:1428] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331: Time spent starting tablet: real 0.242s user 0.020s sys 0.010s
I20260807 03:39:30.470209 18094 tablet_copy_service.cc:342] P d9716e2bf06442f5b3b0e81087c53c4d: Request end of tablet copy session 5c9b5062bdfa46af9396f1a85025a331-6129789267024fac8a337070c10a472a received from {username='slave'} at 127.0.0.1:58950
I20260807 03:39:30.470711 18094 tablet_copy_service.cc:433] P d9716e2bf06442f5b3b0e81087c53c4d: ending tablet copy session 5c9b5062bdfa46af9396f1a85025a331-6129789267024fac8a337070c10a472a on tablet 6129789267024fac8a337070c10a472a with peer 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:39:30.430924 21804 consensus_queue.cc:1035] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
I20260807 03:39:30.474711 21662 consensus_queue.cc:1035] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 03:39:30.475085 21873 raft_consensus.cc:686] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:30.475926 21873 consensus_queue.cc:237] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:30.516593 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 73086.2us
I20260807 03:39:30.537844 18183 raft_consensus.cc:1261] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260807 03:39:30.558786 18081 raft_consensus.cc:1261] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 03:39:30.565112 19297 raft_consensus.cc:1203] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [term 9 LEARNER]: Deduplicated request from leader. Original: 9.5->[9.6-9.6] Dedup: 9.6->[]
I20260807 03:39:30.579612 19297 raft_consensus.cc:2932] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [term 9 LEARNER]: Committing config change with OpId 9.6: config changed from index -1 to 6, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:39:30.576898 21820 consensus_queue.cc:1035] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20260807 03:39:30.599778 21861 raft_consensus.cc:2932] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 6 LEADER]: Committing config change with OpId 6.5: config changed from index -1 to 5, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:39:30.604914 18079 raft_consensus.cc:2932] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Committing config change with OpId 6.5: config changed from index -1 to 5, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:39:30.642630 18183 raft_consensus.cc:1261] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
W20260807 03:39:30.645955 18246 consensus_peers.cc:487] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 -> Peer 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381): Couldn't send request to peer 5c9b5062bdfa46af9396f1a85025a331. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1dbff0b8e97846ea9f567f377b986efc. This is attempt 1: this message will repeat every 5th retry.
I20260807 03:39:30.612658 21770 consensus_queue.cc:1035] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.022s
W20260807 03:39:30.646400 18246 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.132:41381, user_credentials={real_user=slave}} blocked reactor thread for 93501.6us
I20260807 03:39:30.670333 18293 raft_consensus.cc:1261] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260807 03:39:30.657019 21880 consensus_queue.cc:1035] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
I20260807 03:39:30.681941 18298 raft_consensus.cc:1261] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20260807 03:39:30.746819 18246 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.132:41381, user_credentials={real_user=slave}} blocked reactor thread for 79513.1us
I20260807 03:39:30.766637 21824 consensus_queue.cc:1035] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260807 03:39:30.767333 21868 consensus_queue.cc:1035] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 03:39:30.771086 21518 raft_consensus.cc:1050] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d: attempting to promote NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 to VOTER
I20260807 03:39:30.773620 21518 consensus_queue.cc:237] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 2, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:30.773885 18191 raft_consensus.cc:2932] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Committing config change with OpId 6.5: config changed from index -1 to 5, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:39:30.782557 17920 catalog_manager.cc:5495] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 2 to 3, leader changed from d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 3 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:39:30.801604 17914 catalog_manager.cc:5495] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 5 to 6, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 6 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:30.834246 18310 raft_consensus.cc:1261] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 03:39:30.824606 18079 consensus_queue.cc:237] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:39:30.830279 17914 catalog_manager.cc:5495] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: config changed from index -1 to 5, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New cstate: current_term: 6 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:39:30.864046 21792 consensus_queue.cc:1035] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 03:39:30.912689 19295 raft_consensus.cc:1261] T d24d0bbea8bd49d482ad95538ca6e631 P 5c9b5062bdfa46af9396f1a85025a331 [term 4 LEARNER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260807 03:39:30.914752 21824 consensus_queue.cc:1035] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 03:39:30.918665 18183 raft_consensus.cc:1261] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260807 03:39:30.931634 18297 raft_consensus.cc:1261] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260807 03:39:30.932755 21518 consensus_queue.cc:1035] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20260807 03:39:30.939082 21760 consensus_queue.cc:1035] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 03:39:30.947840 21786 ts_tablet_manager.cc:927] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331: Initiating tablet copy from peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:30.967353 21820 raft_consensus.cc:1050] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d: attempting to promote NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 to VOTER
I20260807 03:39:31.030074 18190 raft_consensus.cc:1261] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 03:39:31.043980 21820 consensus_queue.cc:237] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 4, Current term: 9, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:31.027632 21786 tablet_copy_client.cc:316] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Beginning tablet copy session from remote peer at address 127.15.198.131:33021
I20260807 03:39:31.049147 21912 consensus_queue.cc:1035] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20260807 03:39:31.054913 18325 tablet_copy_service.cc:140] P 66a36e1ac8e945e893668e282d394c49: Received BeginTabletCopySession request for tablet 1dbff0b8e97846ea9f567f377b986efc from peer 5c9b5062bdfa46af9396f1a85025a331 ({username='slave'} at 127.0.0.1:37190)
I20260807 03:39:31.055498 18325 tablet_copy_service.cc:161] P 66a36e1ac8e945e893668e282d394c49: Beginning new tablet copy session on tablet 1dbff0b8e97846ea9f567f377b986efc from peer 5c9b5062bdfa46af9396f1a85025a331 at {username='slave'} at 127.0.0.1:37190: session id = 5c9b5062bdfa46af9396f1a85025a331-1dbff0b8e97846ea9f567f377b986efc
W20260807 03:39:31.062960 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 87210.8us
I20260807 03:39:31.066811 18072 raft_consensus.cc:1261] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260807 03:39:31.067948 18312 raft_consensus.cc:1261] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
W20260807 03:39:31.072261 18025 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.132:41381, user_credentials={real_user=slave}} blocked reactor thread for 90275.8us
I20260807 03:39:31.076881 18191 raft_consensus.cc:1261] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260807 03:39:31.074703 21871 consensus_queue.cc:1035] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.013s
I20260807 03:39:31.089572 21848 consensus_queue.cc:1035] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.006s
I20260807 03:39:31.093493 21880 raft_consensus.cc:2932] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 4 to 5, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:31.098665 21884 consensus_queue.cc:1035] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 03:39:30.994889 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260807 03:39:31.105801 18025 consensus_peers.cc:487] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d -> Peer 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381): Couldn't send request to peer 5c9b5062bdfa46af9396f1a85025a331. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4d3a25995c8341c0bf39232774e3a917. This is attempt 1: this message will repeat every 5th retry.
I20260807 03:39:31.109078 19295 raft_consensus.cc:2932] T d24d0bbea8bd49d482ad95538ca6e631 P 5c9b5062bdfa46af9396f1a85025a331 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 4 to 5, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:31.115775 17918 catalog_manager.cc:5495] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 5 to 6, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 6 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:31.120846 18075 raft_consensus.cc:1261] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 03:39:31.122189 21661 consensus_queue.cc:1035] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 03:39:31.123137 18325 tablet_copy_source_session.cc:205] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 03:39:31.133997 18312 raft_consensus.cc:1261] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260807 03:39:31.136422 21798 consensus_queue.cc:1035] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 03:39:31.143147 17914 catalog_manager.cc:5495] T d24d0bbea8bd49d482ad95538ca6e631 P 5c9b5062bdfa46af9396f1a85025a331 reported cstate change: config changed from index 4 to 5, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:31.243432 21786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dbff0b8e97846ea9f567f377b986efc. 1 dirs total, 0 dirs full, 0 dirs failed
W20260807 03:39:31.344646 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 135053us
I20260807 03:39:31.384858 18190 raft_consensus.cc:2932] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 4 to 5, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
W20260807 03:39:31.482389 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 81185us
I20260807 03:39:31.490217 18072 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4ae0265e0707432980672f7cc955c571"
dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
mode: GRACEFUL
new_leader_uuid: "66a36e1ac8e945e893668e282d394c49"
from {username='slave'} at 127.0.0.1:55742
I20260807 03:39:31.496384 18072 raft_consensus.cc:595] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 LEADER]: Received request to transfer leadership to TS 66a36e1ac8e945e893668e282d394c49
W20260807 03:39:31.499338 21955 log.cc:924] Time spent T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.052s user 0.001s sys 0.000s
I20260807 03:39:31.513348 18190 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
mode: GRACEFUL
new_leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:34758
I20260807 03:39:31.514088 18190 raft_consensus.cc:595] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Received request to transfer leadership to TS d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:31.521685 18298 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16"
dest_uuid: "66a36e1ac8e945e893668e282d394c49"
mode: GRACEFUL
new_leader_uuid: "3c4bceb836db4536a0f255fdf86abc02"
from {username='slave'} at 127.0.0.1:58736
I20260807 03:39:31.522270 18298 raft_consensus.cc:595] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Received request to transfer leadership to TS 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:31.523619 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260807 03:39:31.525419 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:39:31.526002 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:39:31.546355 18080 consensus_queue.cc:237] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
W20260807 03:39:31.556361 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 133371us
W20260807 03:39:31.594621 21966 log.cc:924] Time spent T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.079s user 0.001s sys 0.000s
I20260807 03:39:31.608011 19295 raft_consensus.cc:1261] T d24d0bbea8bd49d482ad95538ca6e631 P 5c9b5062bdfa46af9396f1a85025a331 [term 4 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 03:39:31.607108 21971 raft_consensus.cc:2932] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 LEADER]: Committing config change with OpId 6.6: config changed from index -1 to 6, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:39:31.609323 19292 raft_consensus.cc:1261] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [term 9 LEARNER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260807 03:39:31.612273 21959 consensus_queue.cc:1035] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 03:39:31.612726 21518 consensus_queue.cc:1035] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260807 03:39:31.614712 21968 log.cc:924] Time spent T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.178s user 0.001s sys 0.000s
I20260807 03:39:31.631982 21786 tablet_copy_client.cc:791] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Starting download of 0 data blocks...
I20260807 03:39:31.633939 21786 tablet_copy_client.cc:655] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Starting download of 1 WAL segments...
I20260807 03:39:31.636198 18300 raft_consensus.cc:2932] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Committing config change with OpId 6.6: config changed from index -1 to 6, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:39:31.653875 18297 raft_consensus.cc:1261] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260807 03:39:31.656152 18191 raft_consensus.cc:1261] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 03:39:31.660805 18180 raft_consensus.cc:2932] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Committing config change with OpId 6.6: config changed from index -1 to 6, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:39:31.673406 21972 ts_tablet_manager.cc:927] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331: Initiating tablet copy from peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163)
I20260807 03:39:31.693616 21786 tablet_copy_client.cc:531] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 03:39:31.711138 18184 raft_consensus.cc:1261] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260807 03:39:31.710238 19295 raft_consensus.cc:2932] T d24d0bbea8bd49d482ad95538ca6e631 P 5c9b5062bdfa46af9396f1a85025a331 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, VOTER 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:31.713552 21820 raft_consensus.cc:2932] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, VOTER 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:31.716344 21909 raft_consensus.cc:979] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49: : Instructing follower 3c4bceb836db4536a0f255fdf86abc02 to start an election
I20260807 03:39:31.716928 21909 raft_consensus.cc:1067] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Signalling peer 3c4bceb836db4536a0f255fdf86abc02 to start an election
I20260807 03:39:31.717744 21796 raft_consensus.cc:979] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02: : Instructing follower d9716e2bf06442f5b3b0e81087c53c4d to start an election
I20260807 03:39:31.718250 21796 raft_consensus.cc:1067] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Signalling peer d9716e2bf06442f5b3b0e81087c53c4d to start an election
I20260807 03:39:31.726310 18190 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
from {username='slave'} at 127.0.0.1:44660
I20260807 03:39:31.726854 18190 raft_consensus.cc:482] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:39:31.726936 18080 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894"
dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:33268
I20260807 03:39:31.727304 18190 raft_consensus.cc:3037] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:31.727501 18080 raft_consensus.cc:482] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:39:31.727828 18080 raft_consensus.cc:3037] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:31.736756 18080 raft_consensus.cc:504] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:31.738329 18080 leader_election.cc:288] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:31.750366 21871 consensus_queue.cc:1035] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260807 03:39:31.751232 17914 catalog_manager.cc:5495] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 reported cstate change: config changed from index -1 to 6, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New cstate: current_term: 6 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:39:31.760869 18190 raft_consensus.cc:504] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:31.761615 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:31.762287 18191 raft_consensus.cc:3032] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Stepping down as leader of term 6
I20260807 03:39:31.762557 18191 raft_consensus.cc:729] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:31.766633 18191 consensus_queue.cc:260] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:31.767877 18191 raft_consensus.cc:3037] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:31.769336 21786 tablet_bootstrap.cc:492] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331: Bootstrap starting.
I20260807 03:39:31.772647 21972 tablet_copy_client.cc:316] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Beginning tablet copy session from remote peer at address 127.15.198.129:36163
I20260807 03:39:31.777179 18094 tablet_copy_service.cc:140] P d9716e2bf06442f5b3b0e81087c53c4d: Received BeginTabletCopySession request for tablet 4d3a25995c8341c0bf39232774e3a917 from peer 5c9b5062bdfa46af9396f1a85025a331 ({username='slave'} at 127.0.0.1:58950)
I20260807 03:39:31.777733 18094 tablet_copy_service.cc:161] P d9716e2bf06442f5b3b0e81087c53c4d: Beginning new tablet copy session on tablet 4d3a25995c8341c0bf39232774e3a917 from peer 5c9b5062bdfa46af9396f1a85025a331 at {username='slave'} at 127.0.0.1:58950: session id = 5c9b5062bdfa46af9396f1a85025a331-4d3a25995c8341c0bf39232774e3a917
I20260807 03:39:31.783179 17912 catalog_manager.cc:5495] T d24d0bbea8bd49d482ad95538ca6e631 P 5c9b5062bdfa46af9396f1a85025a331 reported cstate change: config changed from index 5 to 6, VOTER 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) evicted. New cstate: current_term: 4 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:31.786870 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:31.787468 18297 raft_consensus.cc:3037] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:31.790772 18191 raft_consensus.cc:2454] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 7.
I20260807 03:39:31.793051 21957 consensus_queue.cc:1035] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 03:39:31.793706 21880 consensus_queue.cc:1035] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260807 03:39:31.798538 18297 raft_consensus.cc:2454] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 7.
I20260807 03:39:31.800802 18190 leader_election.cc:288] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:31.802116 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:31.802878 18072 raft_consensus.cc:3037] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:31.813683 18094 tablet_copy_source_session.cc:205] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 03:39:31.814723 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:31.815281 18300 raft_consensus.cc:3032] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 03:39:31.815590 18300 raft_consensus.cc:729] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:31.816280 18300 consensus_queue.cc:260] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:31.817534 18300 raft_consensus.cc:3037] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:31.820185 21972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d3a25995c8341c0bf39232774e3a917. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:39:31.820887 18027 leader_election.cc:302] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:31.843623 18072 raft_consensus.cc:2454] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 5.
I20260807 03:39:31.848416 21711 raft_consensus.cc:2781] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election won for term 7
I20260807 03:39:31.849002 21711 raft_consensus.cc:686] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:31.849879 21711 consensus_queue.cc:237] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:31.854223 18190 raft_consensus.cc:2932] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, VOTER 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:31.857600 21958 raft_consensus.cc:979] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d: : Instructing follower 66a36e1ac8e945e893668e282d394c49 to start an election
I20260807 03:39:31.858062 18140 leader_election.cc:302] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:31.862763 18300 raft_consensus.cc:2454] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 5.
I20260807 03:39:31.865283 21914 raft_consensus.cc:2781] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 03:39:31.865445 21958 raft_consensus.cc:1067] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 LEADER]: Signalling peer 66a36e1ac8e945e893668e282d394c49 to start an election
I20260807 03:39:31.874617 17898 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet d24d0bbea8bd49d482ad95538ca6e631 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 03:39:31.867158 21914 raft_consensus.cc:686] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:31.879325 18312 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4ae0265e0707432980672f7cc955c571"
dest_uuid: "66a36e1ac8e945e893668e282d394c49"
from {username='slave'} at 127.0.0.1:57276
I20260807 03:39:31.879930 18312 raft_consensus.cc:482] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:39:31.880324 18312 raft_consensus.cc:3037] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:31.877996 21914 consensus_queue.cc:237] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:31.925055 18312 raft_consensus.cc:504] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:31.927853 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:31.929488 18075 raft_consensus.cc:3032] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 LEADER]: Stepping down as leader of term 6
I20260807 03:39:31.929761 18075 raft_consensus.cc:729] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:31.930445 18075 consensus_queue.cc:260] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:31.931464 18075 raft_consensus.cc:3037] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:31.928287 18312 leader_election.cc:288] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:31.936699 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:31.937255 18190 raft_consensus.cc:3037] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:31.955647 18190 raft_consensus.cc:2454] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 7.
I20260807 03:39:31.956836 18249 leader_election.cc:302] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:31.961712 18075 raft_consensus.cc:2454] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 7.
I20260807 03:39:31.961483 21662 raft_consensus.cc:2781] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 03:39:31.984622 21662 raft_consensus.cc:686] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:31.956946 21958 raft_consensus.cc:2932] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 9 LEADER]: Committing config change with OpId 9.7: config changed from index 6 to 7, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:31.999791 21662 consensus_queue.cc:237] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:32.063823 18298 raft_consensus.cc:2932] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Committing config change with OpId 9.7: config changed from index 6 to 7, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:32.072816 18190 raft_consensus.cc:2932] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Committing config change with OpId 9.7: config changed from index 6 to 7, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:32.070297 19295 raft_consensus.cc:2932] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [term 9 FOLLOWER]: Committing config change with OpId 9.7: config changed from index 6 to 7, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
W20260807 03:39:32.082639 21979 log.cc:924] Time spent T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.097s user 0.000s sys 0.001s
I20260807 03:39:32.101368 17920 catalog_manager.cc:5495] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 6 to 7, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 7 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:32.112807 18284 tablet_service.cc:1514] Processing DeleteTablet for tablet d24d0bbea8bd49d482ad95538ca6e631 with delete_type TABLET_DATA_TOMBSTONED (TS 66a36e1ac8e945e893668e282d394c49 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57260
I20260807 03:39:32.121131 17918 catalog_manager.cc:5495] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 reported cstate change: config changed from index 6 to 7, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New cstate: current_term: 9 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:32.131481 17914 catalog_manager.cc:5495] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 4 to 5, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 5 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:32.218068 21972 tablet_copy_client.cc:791] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Starting download of 0 data blocks...
I20260807 03:39:32.219818 21972 tablet_copy_client.cc:655] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Starting download of 1 WAL segments...
I20260807 03:39:32.219089 17912 catalog_manager.cc:5495] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 6 to 7, leader changed from d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 7 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:39:32.229936 21972 tablet_copy_client.cc:531] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 03:39:32.236352 21988 tablet_replica.cc:330] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49: stopping tablet replica
I20260807 03:39:32.237463 21988 raft_consensus.cc:2229] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260807 03:39:32.238118 21988 pending_rounds.cc:62] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49: Trying to abort 1 pending ops.
I20260807 03:39:32.273800 21786 tablet_bootstrap.cc:492] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 03:39:32.273931 18075 consensus_queue.cc:237] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:32.274916 21786 tablet_bootstrap.cc:492] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331: Bootstrap complete.
I20260807 03:39:32.275648 21786 ts_tablet_manager.cc:1397] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331: Time spent bootstrapping tablet: real 0.507s user 0.014s sys 0.028s
I20260807 03:39:32.278434 21786 raft_consensus.cc:348] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331 [term 6 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:39:32.279289 21786 raft_consensus.cc:729] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331 [term 6 LEARNER]: Becoming Follower/Learner. State: Replica: 5c9b5062bdfa46af9396f1a85025a331, State: Initialized, Role: LEARNER
I20260807 03:39:32.279963 21786 consensus_queue.cc:260] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:39:32.282616 21786 ts_tablet_manager.cc:1428] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260807 03:39:32.284373 18325 tablet_copy_service.cc:342] P 66a36e1ac8e945e893668e282d394c49: Request end of tablet copy session 5c9b5062bdfa46af9396f1a85025a331-1dbff0b8e97846ea9f567f377b986efc received from {username='slave'} at 127.0.0.1:37190
I20260807 03:39:32.284757 18325 tablet_copy_service.cc:433] P 66a36e1ac8e945e893668e282d394c49: ending tablet copy session 5c9b5062bdfa46af9396f1a85025a331-1dbff0b8e97846ea9f567f377b986efc on tablet 1dbff0b8e97846ea9f567f377b986efc with peer 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:39:32.287050 19295 raft_consensus.cc:1261] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [term 9 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260807 03:39:32.289876 21958 consensus_queue.cc:1035] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260807 03:39:32.298969 18183 raft_consensus.cc:1261] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260807 03:39:32.301560 21958 consensus_queue.cc:1035] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.007s
I20260807 03:39:32.309998 21820 raft_consensus.cc:2932] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 9 LEADER]: Committing config change with OpId 9.8: config changed from index 7 to 8, VOTER 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:32.320719 19295 raft_consensus.cc:2932] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [term 9 FOLLOWER]: Committing config change with OpId 9.8: config changed from index 7 to 8, VOTER 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:32.324823 21972 tablet_bootstrap.cc:492] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331: Bootstrap starting.
I20260807 03:39:32.326165 18071 raft_consensus.cc:1261] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 03:39:32.327868 21914 consensus_queue.cc:1035] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260807 03:39:32.328313 18183 raft_consensus.cc:2932] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Committing config change with OpId 9.8: config changed from index 7 to 8, VOTER 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:32.238399 21988 pending_rounds.cc:68] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49: Aborting op as it isn't in flight: id { term: 4 index: 5 } timestamp: 7315758984282292224 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } } }
I20260807 03:39:32.332026 21988 raft_consensus.cc:2894] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Aborting config change with OpId 4.5: Aborted: Op aborted
I20260807 03:39:32.332509 21988 raft_consensus.cc:2866] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20260807 03:39:32.332846 21988 raft_consensus.cc:2258] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260807 03:39:32.355572 18191 raft_consensus.cc:1261] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260807 03:39:32.356477 17898 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 6129789267024fac8a337070c10a472a with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 03:39:32.372629 18297 raft_consensus.cc:1261] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 03:39:32.372918 17914 catalog_manager.cc:5495] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 reported cstate change: config changed from index 7 to 8, VOTER 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) evicted. New cstate: current_term: 9 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:32.376835 21848 consensus_queue.cc:1035] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 03:39:32.386468 18284 tablet_service.cc:1514] Processing DeleteTablet for tablet 6129789267024fac8a337070c10a472a with delete_type TABLET_DATA_TOMBSTONED (TS 66a36e1ac8e945e893668e282d394c49 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:57260
I20260807 03:39:32.386809 21988 ts_tablet_manager.cc:1905] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 03:39:32.376835 19295 raft_consensus.cc:1203] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331 [term 6 LEARNER]: Deduplicated request from leader. Original: 6.4->[6.5-6.5] Dedup: 6.5->[]
I20260807 03:39:32.418123 18072 raft_consensus.cc:1261] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260807 03:39:32.419996 21970 consensus_queue.cc:1035] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 03:39:32.423475 18300 raft_consensus.cc:1261] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260807 03:39:32.425905 21518 consensus_queue.cc:1035] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 03:39:32.461998 18183 raft_consensus.cc:1261] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260807 03:39:32.492311 21912 consensus_queue.cc:1035] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 03:39:32.493451 21988 ts_tablet_manager.cc:1918] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.5
I20260807 03:39:32.494004 21988 log.cc:1196] T d24d0bbea8bd49d482ad95538ca6e631 P 66a36e1ac8e945e893668e282d394c49: Deleting WAL directory at /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786073845599630-16154-0/minicluster-data/ts-2-root/wals/d24d0bbea8bd49d482ad95538ca6e631
I20260807 03:39:32.497433 21988 tablet_replica.cc:330] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49: stopping tablet replica
I20260807 03:39:32.496377 21820 consensus_queue.cc:1035] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20260807 03:39:32.542588 21988 raft_consensus.cc:2229] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Raft consensus shutting down.
I20260807 03:39:32.543358 17899 catalog_manager.cc:4841] TS 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021): tablet d24d0bbea8bd49d482ad95538ca6e631 (table test-workload [id=677ca051b50345e183d60556fc42291d]) successfully deleted
W20260807 03:39:32.567909 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 59472.6us
I20260807 03:39:32.601495 21988 raft_consensus.cc:2258] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Raft consensus is shut down!
I20260807 03:39:32.685094 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260807 03:39:32.702471 18025 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.132:41381, user_credentials={real_user=slave}} blocked reactor thread for 63429.6us
W20260807 03:39:32.739147 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 87641.1us
W20260807 03:39:32.782974 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 48402.6us
W20260807 03:39:32.794931 19240 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 94508.3us
I20260807 03:39:32.803164 21988 ts_tablet_manager.cc:1905] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260807 03:39:32.979123 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 142917us
I20260807 03:39:33.010558 18184 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
mode: GRACEFUL
new_leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:34758
I20260807 03:39:33.014276 18184 raft_consensus.cc:595] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Received request to transfer leadership to TS d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:33.033481 21972 tablet_bootstrap.cc:492] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 03:39:33.034379 21972 tablet_bootstrap.cc:492] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331: Bootstrap complete.
I20260807 03:39:33.035017 21972 ts_tablet_manager.cc:1397] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331: Time spent bootstrapping tablet: real 0.710s user 0.032s sys 0.015s
W20260807 03:39:33.056598 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 223217us
W20260807 03:39:33.070510 18025 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.132:41381, user_credentials={real_user=slave}} blocked reactor thread for 116851us
I20260807 03:39:33.055953 21972 raft_consensus.cc:348] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331 [term 6 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:39:33.125916 21972 raft_consensus.cc:729] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331 [term 6 LEARNER]: Becoming Follower/Learner. State: Replica: 5c9b5062bdfa46af9396f1a85025a331, State: Initialized, Role: LEARNER
W20260807 03:39:33.160676 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 136148us
I20260807 03:39:33.462649 18298 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6"
dest_uuid: "66a36e1ac8e945e893668e282d394c49"
mode: GRACEFUL
new_leader_uuid: "3c4bceb836db4536a0f255fdf86abc02"
from {username='slave'} at 127.0.0.1:58736
W20260807 03:39:33.360419 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 74359.5us
I20260807 03:39:33.478128 18298 raft_consensus.cc:595] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Received request to transfer leadership to TS 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:33.499051 21988 ts_tablet_manager.cc:1918] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 9.7
I20260807 03:39:33.499576 21988 log.cc:1196] T 6129789267024fac8a337070c10a472a P 66a36e1ac8e945e893668e282d394c49: Deleting WAL directory at /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786073845599630-16154-0/minicluster-data/ts-2-root/wals/6129789267024fac8a337070c10a472a
I20260807 03:39:33.530157 21988 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57260 (request call id 364) took 1135 ms. Trace:
I20260807 03:39:33.530612 21988 rpcz_store.cc:276] 0807 03:39:32.385420 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:32.386250 (+ 830us) service_pool.cc:227] Handling call
0807 03:39:32.497091 (+110841us) ts_tablet_manager.cc:1145] Deleting tablet 6129789267024fac8a337070c10a472a
0807 03:39:32.497149 (+ 58us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0807 03:39:33.041291 (+544142us) tablet_metadata.cc:641] Metadata flushed
0807 03:39:33.498768 (+457477us) tablet_metadata.cc:641] Metadata flushed
0807 03:39:33.518379 (+ 19611us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22289,"tablet-delete.queue_time_us":110072}
I20260807 03:39:33.544747 22050 raft_consensus.cc:979] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49: : Instructing follower 3c4bceb836db4536a0f255fdf86abc02 to start an election
I20260807 03:39:33.545445 22050 raft_consensus.cc:1067] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Signalling peer 3c4bceb836db4536a0f255fdf86abc02 to start an election
I20260807 03:39:33.545696 21861 raft_consensus.cc:1050] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49: attempting to promote NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 to VOTER
W20260807 03:39:33.545995 22054 log.cc:924] Time spent T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.208s user 0.003s sys 0.008s
I20260807 03:39:33.548290 21861 consensus_queue.cc:237] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 3, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
W20260807 03:39:33.566378 22002 log.cc:924] Time spent T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.228s user 0.003s sys 0.006s
I20260807 03:39:33.571542 18183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44660 (request call id 1998) took 1075 ms. Trace:
I20260807 03:39:33.571991 18183 rpcz_store.cc:276] 0807 03:39:32.496238 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:32.501910 (+ 5672us) service_pool.cc:227] Handling call
0807 03:39:32.502275 (+ 365us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:32.675939 (+173664us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:32.675984 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:39:32.676556 (+ 572us) log.cc:842] Serialized 26 byte log entry
0807 03:39:33.157848 (+481292us) raft_consensus.cc:1631] Marking committed up to 6
0807 03:39:33.483787 (+325939us) log.cc:842] Serialized 14 byte log entry
0807 03:39:33.484057 (+ 270us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:33.484174 (+ 117us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:33.545717 (+ 61543us) spinlock_profiling.cc:243] Waited 27.8 ms on lock 0x7b2400138428. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f0efcf4 00007fe98f0eddfb 00007fe98f17c559 00007fe98f17c1e5 00007fe98f17b82d 00007fe991e2a46e 00007fe991e41c0c 00007fe991e40f36 00007fe991e40b58 00007fe991e4a551
0807 03:39:33.571219 (+ 25502us) raft_consensus.cc:1671] finished
0807 03:39:33.571323 (+ 104us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:33.571457 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11323,"prepare.queue_time_us":340898,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":32,"raft.queue_time_us":2806,"raft.run_cpu_time_us":263,"raft.run_wall_time_us":38772,"spinlock_wait_cycles":63923584,"thread_start_us":738543,"threads_started":3,"wal-append.queue_time_us":609504}
I20260807 03:39:33.575438 17899 catalog_manager.cc:4841] TS 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021): tablet 6129789267024fac8a337070c10a472a (table test-workload [id=677ca051b50345e183d60556fc42291d]) successfully deleted
I20260807 03:39:33.588279 18069 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964"
dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
mode: GRACEFUL
new_leader_uuid: "66a36e1ac8e945e893668e282d394c49"
from {username='slave'} at 127.0.0.1:55742
I20260807 03:39:33.590111 18069 raft_consensus.cc:595] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 LEADER]: Received request to transfer leadership to TS 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:33.601572 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260807 03:39:33.603884 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:39:33.604779 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:39:33.255402 21972 consensus_queue.cc:260] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:39:33.667250 21972 ts_tablet_manager.cc:1428] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331: Time spent starting tablet: real 0.632s user 0.009s sys 0.011s
W20260807 03:39:33.667618 21972 ts_tablet_manager.cc:1460] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331: Tablet startup took 632ms
W20260807 03:39:33.667951 21972 ts_tablet_manager.cc:1462] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331: Trace:
0807 03:39:31.606590 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:31.606753 (+ 163us) service_pool.cc:227] Handling call
0807 03:39:31.673740 (+ 66987us) ts_tablet_manager.cc:928] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331: Initiating tablet copy from peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163)
0807 03:39:31.818801 (+145061us) tablet_copy_client.cc:361] Tablet copy session begun
0807 03:39:31.892893 (+ 74092us) tablet_metadata.cc:641] Metadata flushed
0807 03:39:31.893296 (+ 403us) tablet_copy_client.cc:490] Wrote new tablet metadata
0807 03:39:32.117421 (+224125us) tablet_copy_client.cc:411] Wrote new consensus metadata
0807 03:39:32.182134 (+ 64713us) tablet_replica.cc:205] Creating consensus instance
0807 03:39:32.217453 (+ 35319us) ts_tablet_manager.cc:983] Replica registered.
0807 03:39:32.217546 (+ 93us) inbound_call.cc:173] Queueing success response
0807 03:39:32.324490 (+106944us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0807 03:39:33.035292 (+710802us) ts_tablet_manager.cc:1429] Starting tablet replica
0807 03:39:33.039154 (+ 3862us) tablet_replica.cc:259] Starting instrumentation
0807 03:39:33.039716 (+ 562us) tablet_replica.cc:278] Starting consensus
Metrics: {"dns_us":42045,"raft.queue_time_us":4260,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":4734,"tablet-copy.queue_time_us":66133,"tablet-download-pool.queue_time_us":112,"tablet-download-pool.run_cpu_time_us":1187,"tablet-download-pool.run_wall_time_us":4332,"thread_start_us":71245,"threads_started":2}
I20260807 03:39:33.672429 18094 tablet_copy_service.cc:342] P d9716e2bf06442f5b3b0e81087c53c4d: Request end of tablet copy session 5c9b5062bdfa46af9396f1a85025a331-4d3a25995c8341c0bf39232774e3a917 received from {username='slave'} at 127.0.0.1:58950
I20260807 03:39:33.672830 18094 tablet_copy_service.cc:433] P d9716e2bf06442f5b3b0e81087c53c4d: ending tablet copy session 5c9b5062bdfa46af9396f1a85025a331-4d3a25995c8341c0bf39232774e3a917 on tablet 4d3a25995c8341c0bf39232774e3a917 with peer 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:39:33.681794 18187 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
from {username='slave'} at 127.0.0.1:44660
I20260807 03:39:33.682444 18187 raft_consensus.cc:482] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:39:33.685261 18187 raft_consensus.cc:3037] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Advancing to term 8
W20260807 03:39:33.696619 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 66643.4us
I20260807 03:39:33.701932 18187 raft_consensus.cc:504] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:33.707021 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 54255.4us
I20260807 03:39:33.708433 18180 raft_consensus.cc:1261] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
W20260807 03:39:33.708462 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 158154us
I20260807 03:39:33.710739 22055 raft_consensus.cc:979] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02: : Instructing follower d9716e2bf06442f5b3b0e81087c53c4d to start an election
I20260807 03:39:33.715016 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:33.715619 18310 raft_consensus.cc:3032] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Stepping down as leader of term 7
I20260807 03:39:33.715888 18310 raft_consensus.cc:729] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:33.716572 18310 consensus_queue.cc:260] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:33.717639 18310 raft_consensus.cc:3037] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:33.726637 22055 raft_consensus.cc:1067] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Signalling peer d9716e2bf06442f5b3b0e81087c53c4d to start an election
I20260807 03:39:33.707126 18187 leader_election.cc:288] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:33.740355 19297 raft_consensus.cc:1261] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331 [term 6 LEARNER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260807 03:39:33.742555 22076 consensus_queue.cc:1035] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 03:39:33.746452 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:33.747025 18069 raft_consensus.cc:3037] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:33.744879 21861 consensus_queue.cc:1035] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 03:39:33.753456 18080 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf"
dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:33268
I20260807 03:39:33.754022 18080 raft_consensus.cc:482] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:39:33.754328 18080 raft_consensus.cc:3037] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:33.759279 18069 raft_consensus.cc:2454] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 8.
I20260807 03:39:33.760577 18140 leader_election.cc:302] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:33.761423 22024 raft_consensus.cc:2781] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader election won for term 8
I20260807 03:39:33.761935 22024 raft_consensus.cc:686] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:33.762827 22024 consensus_queue.cc:237] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:33.766322 18080 raft_consensus.cc:504] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:33.735211 18310 raft_consensus.cc:2454] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 8.
I20260807 03:39:33.794832 18080 leader_election.cc:288] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:33.802346 17912 catalog_manager.cc:5495] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 7 to 8, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 8 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:33.805667 18069 raft_consensus.cc:1261] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260807 03:39:33.806303 19295 raft_consensus.cc:1203] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331 [term 6 LEARNER]: Deduplicated request from leader. Original: 6.5->[6.6-6.6] Dedup: 6.6->[]
I20260807 03:39:33.810189 22083 consensus_queue.cc:1035] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260807 03:39:33.852170 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RunLeaderElection -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 77649.8us
I20260807 03:39:33.890683 22067 raft_consensus.cc:2932] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 6 LEADER]: Committing config change with OpId 6.6: config changed from index 5 to 6, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:33.898926 18072 raft_consensus.cc:2932] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Committing config change with OpId 6.6: config changed from index 5 to 6, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:33.911202 18183 raft_consensus.cc:2932] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Committing config change with OpId 6.6: config changed from index 5 to 6, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:33.936035 22053 raft_consensus.cc:979] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d: : Instructing follower 66a36e1ac8e945e893668e282d394c49 to start an election
I20260807 03:39:33.936650 22053 raft_consensus.cc:1067] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 LEADER]: Signalling peer 66a36e1ac8e945e893668e282d394c49 to start an election
I20260807 03:39:33.938177 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:33.940970 18312 raft_consensus.cc:3037] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:34.014488 18312 raft_consensus.cc:2454] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 5.
I20260807 03:39:34.016810 19297 raft_consensus.cc:2932] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331 [term 6 FOLLOWER]: Committing config change with OpId 6.6: config changed from index 5 to 6, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:34.054109 18072 raft_consensus.cc:1226] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 4. Current term is 5. Ops: []
I20260807 03:39:34.056712 22040 consensus_queue.cc:1046] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 03:39:34.057902 22024 raft_consensus.cc:3032] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 03:39:34.058362 22024 raft_consensus.cc:729] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:34.059191 22024 consensus_queue.cc:260] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:34.060648 22024 raft_consensus.cc:3037] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:34.083096 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:34.085106 18312 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964"
dest_uuid: "66a36e1ac8e945e893668e282d394c49"
from {username='slave'} at 127.0.0.1:57276
I20260807 03:39:34.085705 18312 raft_consensus.cc:482] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:39:34.086170 18312 raft_consensus.cc:3037] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:34.086664 18027 leader_election.cc:302] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:34.087538 22048 raft_consensus.cc:2781] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election won for term 5
I20260807 03:39:34.088027 22048 raft_consensus.cc:686] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:34.088852 22048 consensus_queue.cc:237] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:34.104954 18187 raft_consensus.cc:2454] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 5.
I20260807 03:39:34.112775 18312 raft_consensus.cc:504] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:34.124246 18312 leader_election.cc:288] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:34.124408 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: true dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:34.125061 18072 raft_consensus.cc:3032] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 LEADER]: Stepping down as leader of term 7
I20260807 03:39:34.125341 18072 raft_consensus.cc:729] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:34.127691 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: true dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:34.128290 18187 raft_consensus.cc:3037] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:34.131469 17920 catalog_manager.cc:5495] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 reported cstate change: config changed from index 5 to 6, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New cstate: current_term: 6 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 03:39:34.126544 18072 consensus_queue.cc:260] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:34.138087 18187 raft_consensus.cc:2454] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 8.
I20260807 03:39:34.139434 18249 leader_election.cc:302] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:34.139722 18072 raft_consensus.cc:3037] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:34.140478 21861 raft_consensus.cc:2781] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader election won for term 8
I20260807 03:39:34.141038 21861 raft_consensus.cc:686] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 8 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:34.141973 21861 consensus_queue.cc:237] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:34.174173 18072 raft_consensus.cc:2454] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 8.
I20260807 03:39:34.184481 18309 raft_consensus.cc:1226] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 8 LEADER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 7. Current term is 8. Ops: []
I20260807 03:39:34.225049 18072 raft_consensus.cc:1261] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260807 03:39:34.226277 22005 consensus_queue.cc:1035] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.005s
I20260807 03:39:34.285236 18300 consensus_queue.cc:237] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:34.287829 18312 raft_consensus.cc:1261] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260807 03:39:34.289469 22011 consensus_queue.cc:1035] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 03:39:34.294801 18187 raft_consensus.cc:1261] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260807 03:39:34.294842 19297 raft_consensus.cc:1261] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331 [term 6 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260807 03:39:34.296463 22071 consensus_queue.cc:1035] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.004s
I20260807 03:39:34.301867 21912 consensus_queue.cc:1035] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260807 03:39:34.316993 22058 raft_consensus.cc:2932] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 6 LEADER]: Committing config change with OpId 6.7: config changed from index 6 to 7, VOTER d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:34.330756 18183 raft_consensus.cc:2932] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Committing config change with OpId 6.7: config changed from index 6 to 7, VOTER d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:34.337903 19297 raft_consensus.cc:2932] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331 [term 6 FOLLOWER]: Committing config change with OpId 6.7: config changed from index 6 to 7, VOTER d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:34.380220 17914 catalog_manager.cc:5495] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 4 to 5, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 5 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:34.387548 17918 catalog_manager.cc:5495] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 7 to 8, leader changed from d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 8 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:39:34.402284 21871 raft_consensus.cc:1050] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d: attempting to promote NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 to VOTER
I20260807 03:39:34.407152 21871 consensus_queue.cc:237] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 4, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:34.421770 19297 raft_consensus.cc:1261] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331 [term 6 LEARNER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260807 03:39:34.423213 22041 consensus_queue.cc:1035] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260807 03:39:34.429347 17912 catalog_manager.cc:5495] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331 reported cstate change: config changed from index 6 to 7, VOTER d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) evicted. New cstate: current_term: 6 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:34.430169 18300 raft_consensus.cc:1261] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260807 03:39:34.436492 17899 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 1dbff0b8e97846ea9f567f377b986efc with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 03:39:34.437569 18187 raft_consensus.cc:1261] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260807 03:39:34.454047 21958 consensus_queue.cc:1035] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.017s
I20260807 03:39:34.467142 21871 consensus_queue.cc:1035] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260807 03:39:34.510862 18064 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dbff0b8e97846ea9f567f377b986efc with delete_type TABLET_DATA_TOMBSTONED (TS d9716e2bf06442f5b3b0e81087c53c4d not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33238
I20260807 03:39:34.542430 22116 tablet_replica.cc:330] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d: stopping tablet replica
I20260807 03:39:34.538237 18187 raft_consensus.cc:2932] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Committing config change with OpId 6.7: config changed from index 6 to 7, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:34.543671 22116 raft_consensus.cc:2229] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Raft consensus shutting down.
I20260807 03:39:34.544502 22116 raft_consensus.cc:2258] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Raft consensus is shut down!
I20260807 03:39:34.535717 21957 raft_consensus.cc:2932] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 LEADER]: Committing config change with OpId 6.7: config changed from index 6 to 7, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:34.573499 22116 ts_tablet_manager.cc:1905] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 03:39:34.576094 17918 catalog_manager.cc:5495] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: config changed from index 6 to 7, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New cstate: current_term: 6 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:34.593279 19297 raft_consensus.cc:2932] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331 [term 6 FOLLOWER]: Committing config change with OpId 6.7: config changed from index 6 to 7, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:34.655370 18300 raft_consensus.cc:2932] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Committing config change with OpId 6.7: config changed from index 6 to 7, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:34.663214 18069 raft_consensus.cc:1261] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260807 03:39:34.669524 22116 ts_tablet_manager.cc:1918] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 6.6
I20260807 03:39:34.680420 22116 log.cc:1196] T 1dbff0b8e97846ea9f567f377b986efc P d9716e2bf06442f5b3b0e81087c53c4d: Deleting WAL directory at /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786073845599630-16154-0/minicluster-data/ts-0-root/wals/1dbff0b8e97846ea9f567f377b986efc
I20260807 03:39:34.680701 18187 raft_consensus.cc:1261] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 03:39:34.671429 18183 raft_consensus.cc:1261] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260807 03:39:34.675650 18072 consensus_queue.cc:237] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:34.683444 21871 consensus_queue.cc:1035] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:39:34.688365 22071 consensus_queue.cc:1035] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 03:39:34.691985 22071 consensus_queue.cc:1035] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 03:39:34.692868 19297 raft_consensus.cc:1261] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331 [term 6 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260807 03:39:34.694181 18297 raft_consensus.cc:1261] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 03:39:34.696569 17898 catalog_manager.cc:4841] TS d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): tablet 1dbff0b8e97846ea9f567f377b986efc (table test-workload [id=677ca051b50345e183d60556fc42291d]) successfully deleted
I20260807 03:39:34.697155 21957 consensus_queue.cc:1035] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260807 03:39:34.704434 18312 raft_consensus.cc:1261] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260807 03:39:34.705941 21958 consensus_queue.cc:1035] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:39:34.707592 21958 consensus_queue.cc:1035] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260807 03:39:34.730072 22041 raft_consensus.cc:2932] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 LEADER]: Committing config change with OpId 6.8: config changed from index 7 to 8, VOTER 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:34.743819 18297 raft_consensus.cc:2932] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Committing config change with OpId 6.8: config changed from index 7 to 8, VOTER 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:34.745234 19297 raft_consensus.cc:2932] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331 [term 6 FOLLOWER]: Committing config change with OpId 6.8: config changed from index 7 to 8, VOTER 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:34.752265 17898 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 4d3a25995c8341c0bf39232774e3a917 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 03:39:34.814203 17912 catalog_manager.cc:5495] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331 reported cstate change: config changed from index 7 to 8, VOTER 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) evicted. New cstate: current_term: 6 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:34.919727 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260807 03:39:34.978354 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 124423us
I20260807 03:39:34.999557 18168 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d3a25995c8341c0bf39232774e3a917 with delete_type TABLET_DATA_TOMBSTONED (TS 3c4bceb836db4536a0f255fdf86abc02 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:44650
I20260807 03:39:35.073935 22134 tablet_replica.cc:330] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02: stopping tablet replica
I20260807 03:39:35.064805 18072 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616"
dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
mode: GRACEFUL
new_leader_uuid: "66a36e1ac8e945e893668e282d394c49"
from {username='slave'} at 127.0.0.1:55742
I20260807 03:39:35.104650 18072 raft_consensus.cc:595] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 LEADER]: Received request to transfer leadership to TS 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:35.123194 18187 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
mode: GRACEFUL
new_leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:34758
I20260807 03:39:35.123842 18187 raft_consensus.cc:595] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Received request to transfer leadership to TS d9716e2bf06442f5b3b0e81087c53c4d
W20260807 03:39:35.185086 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 62338.8us
W20260807 03:39:35.197068 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 45799.6us
I20260807 03:39:35.215673 18297 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8c98c1d422654739af06653a512f7499"
dest_uuid: "66a36e1ac8e945e893668e282d394c49"
mode: GRACEFUL
new_leader_uuid: "3c4bceb836db4536a0f255fdf86abc02"
from {username='slave'} at 127.0.0.1:58736
I20260807 03:39:35.216351 18297 raft_consensus.cc:595] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Received request to transfer leadership to TS 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:35.238528 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260807 03:39:35.242278 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:39:35.243067 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:39:35.215991 22134 raft_consensus.cc:2229] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Raft consensus shutting down.
I20260807 03:39:35.261811 22134 raft_consensus.cc:2258] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Raft consensus is shut down!
I20260807 03:39:35.295645 22134 ts_tablet_manager.cc:1905] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260807 03:39:35.338554 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 142952us
I20260807 03:39:35.401113 22132 raft_consensus.cc:979] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d: : Instructing follower 66a36e1ac8e945e893668e282d394c49 to start an election
I20260807 03:39:35.401589 22132 raft_consensus.cc:1067] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 LEADER]: Signalling peer 66a36e1ac8e945e893668e282d394c49 to start an election
I20260807 03:39:35.402393 22134 ts_tablet_manager.cc:1918] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 6.7
I20260807 03:39:35.402917 22134 log.cc:1196] T 4d3a25995c8341c0bf39232774e3a917 P 3c4bceb836db4536a0f255fdf86abc02: Deleting WAL directory at /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786073845599630-16154-0/minicluster-data/ts-1-root/wals/4d3a25995c8341c0bf39232774e3a917
I20260807 03:39:35.404505 17899 catalog_manager.cc:4841] TS 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057): tablet 4d3a25995c8341c0bf39232774e3a917 (table test-workload [id=677ca051b50345e183d60556fc42291d]) successfully deleted
I20260807 03:39:35.423218 21848 raft_consensus.cc:979] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02: : Instructing follower d9716e2bf06442f5b3b0e81087c53c4d to start an election
I20260807 03:39:35.423748 21848 raft_consensus.cc:1067] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Signalling peer d9716e2bf06442f5b3b0e81087c53c4d to start an election
I20260807 03:39:35.431510 18309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616"
dest_uuid: "66a36e1ac8e945e893668e282d394c49"
from {username='slave'} at 127.0.0.1:57276
I20260807 03:39:35.432129 18309 raft_consensus.cc:482] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:39:35.432473 18309 raft_consensus.cc:3037] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:35.444535 18309 raft_consensus.cc:504] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:35.446504 18309 leader_election.cc:288] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:35.447592 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:35.448199 18188 raft_consensus.cc:3037] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:35.448238 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:35.448937 18075 raft_consensus.cc:3032] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 LEADER]: Stepping down as leader of term 4
I20260807 03:39:35.449247 18075 raft_consensus.cc:729] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:35.450006 18075 consensus_queue.cc:260] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:35.451107 18075 raft_consensus.cc:3037] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:35.455122 18188 raft_consensus.cc:2454] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:35.464329 18069 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7"
dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:33268
I20260807 03:39:35.464609 18249 leader_election.cc:302] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:35.464970 18069 raft_consensus.cc:482] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:39:35.465340 18069 raft_consensus.cc:3037] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:35.466329 22124 raft_consensus.cc:2781] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 03:39:35.466876 22124 raft_consensus.cc:686] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:35.467773 22124 consensus_queue.cc:237] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:35.471022 18075 raft_consensus.cc:2454] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:35.472235 18069 raft_consensus.cc:504] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:35.474311 18069 leader_election.cc:288] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:35.480449 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:35.481092 18190 raft_consensus.cc:3032] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Stepping down as leader of term 6
I20260807 03:39:35.481396 18190 raft_consensus.cc:729] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:35.482147 18190 consensus_queue.cc:260] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:35.483253 18190 raft_consensus.cc:3037] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:35.491189 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:35.491771 18312 raft_consensus.cc:3037] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:35.498273 18190 raft_consensus.cc:2454] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 7.
I20260807 03:39:35.499933 18027 leader_election.cc:302] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:35.500491 18312 raft_consensus.cc:2454] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 7.
I20260807 03:39:35.501109 22147 raft_consensus.cc:2781] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election won for term 7
I20260807 03:39:35.501643 22147 raft_consensus.cc:686] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:35.503660 22147 consensus_queue.cc:237] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:35.512810 17920 catalog_manager.cc:5495] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 4 to 5, leader changed from d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 5 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:39:35.529628 22123 raft_consensus.cc:979] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49: : Instructing follower 3c4bceb836db4536a0f255fdf86abc02 to start an election
I20260807 03:39:35.532720 22124 raft_consensus.cc:1067] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Signalling peer 3c4bceb836db4536a0f255fdf86abc02 to start an election
I20260807 03:39:35.549762 17918 catalog_manager.cc:5495] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 6 to 7, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 7 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:35.554064 18188 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c98c1d422654739af06653a512f7499"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
from {username='slave'} at 127.0.0.1:44660
I20260807 03:39:35.554678 18188 raft_consensus.cc:482] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:39:35.555182 18188 raft_consensus.cc:3037] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:35.561152 18188 raft_consensus.cc:504] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:35.563443 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:35.563992 18069 raft_consensus.cc:3037] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:35.570523 18069 raft_consensus.cc:2454] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 6.
I20260807 03:39:35.571699 18140 leader_election.cc:302] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:35.572502 22165 raft_consensus.cc:2781] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 03:39:35.573135 18188 leader_election.cc:288] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:35.573123 22165 raft_consensus.cc:686] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:35.574303 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:35.574864 18309 raft_consensus.cc:3032] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 03:39:35.575254 18309 raft_consensus.cc:729] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:35.574864 22165 consensus_queue.cc:237] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:35.576054 18309 consensus_queue.cc:260] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:35.577437 18309 raft_consensus.cc:3037] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:35.609216 18309 raft_consensus.cc:2454] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 6.
I20260807 03:39:35.670519 17920 catalog_manager.cc:5495] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 5 to 6, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 6 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
W20260807 03:39:35.893895 18246 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.132:41381, user_credentials={real_user=slave}} blocked reactor thread for 96117.1us
W20260807 03:39:35.910128 18025 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.132:41381, user_credentials={real_user=slave}} blocked reactor thread for 65261.2us
I20260807 03:39:35.945377 18188 raft_consensus.cc:1261] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260807 03:39:35.946542 21957 consensus_queue.cc:1035] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 03:39:35.967329 18190 consensus_queue.cc:237] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:39:35.988487 18069 raft_consensus.cc:1261] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 03:39:35.990202 22124 consensus_queue.cc:1035] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:39:36.007763 18309 raft_consensus.cc:1261] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260807 03:39:36.011246 21848 consensus_queue.cc:1035] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
W20260807 03:39:36.013861 18138 consensus_peers.cc:487] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 -> Peer 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381): Couldn't send request to peer 5c9b5062bdfa46af9396f1a85025a331. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a21f43f2cbf24aeca08087d9e2fb9b2a. This is attempt 1: this message will repeat every 5th retry.
I20260807 03:39:36.015013 18190 raft_consensus.cc:1261] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 03:39:36.017423 22180 consensus_queue.cc:1035] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:39:36.022176 18075 raft_consensus.cc:1261] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260807 03:39:36.023972 22165 consensus_queue.cc:1035] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 03:39:36.042146 18300 raft_consensus.cc:1261] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260807 03:39:36.045100 21957 consensus_queue.cc:1035] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 03:39:36.119392 21848 raft_consensus.cc:2932] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Committing config change with OpId 6.5: config changed from index -1 to 5, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:39:36.145668 18312 raft_consensus.cc:2932] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Committing config change with OpId 6.5: config changed from index -1 to 5, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
W20260807 03:39:36.167757 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 49244.3us
W20260807 03:39:36.171324 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 51619.4us
I20260807 03:39:36.203718 18069 raft_consensus.cc:1261] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 03:39:36.211931 22210 consensus_queue.cc:1035] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:39:36.213793 18293 raft_consensus.cc:1261] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 03:39:36.218753 22204 consensus_queue.cc:1035] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:39:36.253633 18080 raft_consensus.cc:2932] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Committing config change with OpId 6.5: config changed from index -1 to 5, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:39:36.257218 18075 consensus_queue.cc:237] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:39:36.275525 17918 catalog_manager.cc:5495] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 reported cstate change: config changed from index -1 to 5, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New cstate: current_term: 6 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:39:36.329622 18187 raft_consensus.cc:1261] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 03:39:36.331159 21957 consensus_queue.cc:1035] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260807 03:39:36.334789 18025 consensus_peers.cc:487] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d -> Peer 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381): Couldn't send request to peer 5c9b5062bdfa46af9396f1a85025a331. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a3dc635ee56b4559a87f5955bb83235a. This is attempt 1: this message will repeat every 5th retry.
I20260807 03:39:36.344715 18312 raft_consensus.cc:1261] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 03:39:36.359491 22211 consensus_queue.cc:1035] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20260807 03:39:36.366997 21957 raft_consensus.cc:2932] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:39:36.411712 18293 raft_consensus.cc:2932] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:39:36.419553 18187 raft_consensus.cc:2932] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:39:36.433419 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:39:36.441919 18080 consensus_queue.cc:237] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:39:36.468887 17918 catalog_manager.cc:5495] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: config changed from index -1 to 3, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New cstate: current_term: 3 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:36.483060 18293 raft_consensus.cc:1261] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260807 03:39:36.507515 22132 consensus_queue.cc:1035] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 03:39:36.511636 22231 ts_tablet_manager.cc:927] T a21f43f2cbf24aeca08087d9e2fb9b2a P 5c9b5062bdfa46af9396f1a85025a331: Initiating tablet copy from peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:36.516703 18080 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88"
dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
mode: GRACEFUL
new_leader_uuid: "66a36e1ac8e945e893668e282d394c49"
from {username='slave'} at 127.0.0.1:55742
I20260807 03:39:36.517351 18080 raft_consensus.cc:595] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 LEADER]: Received request to transfer leadership to TS 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:36.523540 22231 tablet_copy_client.cc:316] T a21f43f2cbf24aeca08087d9e2fb9b2a P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Beginning tablet copy session from remote peer at address 127.15.198.130:37057
I20260807 03:39:36.529000 18207 tablet_copy_service.cc:140] P 3c4bceb836db4536a0f255fdf86abc02: Received BeginTabletCopySession request for tablet a21f43f2cbf24aeca08087d9e2fb9b2a from peer 5c9b5062bdfa46af9396f1a85025a331 ({username='slave'} at 127.0.0.1:50060)
I20260807 03:39:36.529567 18207 tablet_copy_service.cc:161] P 3c4bceb836db4536a0f255fdf86abc02: Beginning new tablet copy session on tablet a21f43f2cbf24aeca08087d9e2fb9b2a from peer 5c9b5062bdfa46af9396f1a85025a331 at {username='slave'} at 127.0.0.1:50060: session id = 5c9b5062bdfa46af9396f1a85025a331-a21f43f2cbf24aeca08087d9e2fb9b2a
I20260807 03:39:36.532796 18187 raft_consensus.cc:1261] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
W20260807 03:39:36.534159 18025 consensus_peers.cc:487] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d -> Peer 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381): Couldn't send request to peer 5c9b5062bdfa46af9396f1a85025a331. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b15fdfae36e94837adea3c6b66864b0e. This is attempt 1: this message will repeat every 5th retry.
I20260807 03:39:36.540126 18207 tablet_copy_source_session.cc:205] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 03:39:36.540278 18188 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
mode: GRACEFUL
new_leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:34758
I20260807 03:39:36.555279 18188 raft_consensus.cc:595] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 LEADER]: Received request to transfer leadership to TS d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:36.547976 21957 consensus_queue.cc:1035] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 03:39:36.558362 18293 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00"
dest_uuid: "66a36e1ac8e945e893668e282d394c49"
mode: GRACEFUL
new_leader_uuid: "3c4bceb836db4536a0f255fdf86abc02"
from {username='slave'} at 127.0.0.1:58736
I20260807 03:39:36.558961 18293 raft_consensus.cc:595] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Received request to transfer leadership to TS 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:36.560351 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260807 03:39:36.563421 22231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a21f43f2cbf24aeca08087d9e2fb9b2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:39:36.593685 22231 tablet_copy_client.cc:791] T a21f43f2cbf24aeca08087d9e2fb9b2a P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Starting download of 0 data blocks...
I20260807 03:39:36.594429 22231 tablet_copy_client.cc:655] T a21f43f2cbf24aeca08087d9e2fb9b2a P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Starting download of 1 WAL segments...
I20260807 03:39:36.605754 22231 tablet_copy_client.cc:531] T a21f43f2cbf24aeca08087d9e2fb9b2a P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 03:39:36.615383 22231 tablet_bootstrap.cc:492] T a21f43f2cbf24aeca08087d9e2fb9b2a P 5c9b5062bdfa46af9396f1a85025a331: Bootstrap starting.
I20260807 03:39:36.624677 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:39:36.625500 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:39:36.638016 22211 raft_consensus.cc:2932] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Committing config change with OpId 5.5: config changed from index -1 to 5, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:39:36.678568 18300 raft_consensus.cc:2932] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Committing config change with OpId 5.5: config changed from index -1 to 5, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:39:36.689924 18188 raft_consensus.cc:2932] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Committing config change with OpId 5.5: config changed from index -1 to 5, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:39:36.720170 18072 consensus_queue.cc:237] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:39:36.698815 17918 catalog_manager.cc:5495] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 reported cstate change: config changed from index -1 to 5, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New cstate: current_term: 5 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:39:36.765838 18300 raft_consensus.cc:1261] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
W20260807 03:39:36.766476 18025 consensus_peers.cc:487] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d -> Peer 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381): Couldn't send request to peer 5c9b5062bdfa46af9396f1a85025a331. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d8d26ed9a1e340ce8c2d127ac7fbf7f7. This is attempt 1: this message will repeat every 5th retry.
I20260807 03:39:36.768455 18187 raft_consensus.cc:1261] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260807 03:39:36.771968 22239 consensus_queue.cc:1035] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260807 03:39:36.774086 22239 consensus_queue.cc:1035] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260807 03:39:36.818064 22246 ts_tablet_manager.cc:927] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331: Initiating tablet copy from peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163)
I20260807 03:39:36.826233 22246 tablet_copy_client.cc:316] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Beginning tablet copy session from remote peer at address 127.15.198.129:36163
I20260807 03:39:36.827936 18094 tablet_copy_service.cc:140] P d9716e2bf06442f5b3b0e81087c53c4d: Received BeginTabletCopySession request for tablet a3dc635ee56b4559a87f5955bb83235a from peer 5c9b5062bdfa46af9396f1a85025a331 ({username='slave'} at 127.0.0.1:58950)
I20260807 03:39:36.828606 18094 tablet_copy_service.cc:161] P d9716e2bf06442f5b3b0e81087c53c4d: Beginning new tablet copy session on tablet a3dc635ee56b4559a87f5955bb83235a from peer 5c9b5062bdfa46af9396f1a85025a331 at {username='slave'} at 127.0.0.1:58950: session id = 5c9b5062bdfa46af9396f1a85025a331-a3dc635ee56b4559a87f5955bb83235a
I20260807 03:39:36.831472 22211 raft_consensus.cc:2932] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 LEADER]: Committing config change with OpId 9.8: config changed from index -1 to 8, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:39:36.839788 18094 tablet_copy_source_session.cc:205] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 03:39:36.841961 18309 raft_consensus.cc:2932] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Committing config change with OpId 9.8: config changed from index -1 to 8, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:39:36.849259 22231 tablet_bootstrap.cc:492] T a21f43f2cbf24aeca08087d9e2fb9b2a P 5c9b5062bdfa46af9396f1a85025a331: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 03:39:36.850265 22231 tablet_bootstrap.cc:492] T a21f43f2cbf24aeca08087d9e2fb9b2a P 5c9b5062bdfa46af9396f1a85025a331: Bootstrap complete.
I20260807 03:39:36.851254 22231 ts_tablet_manager.cc:1397] T a21f43f2cbf24aeca08087d9e2fb9b2a P 5c9b5062bdfa46af9396f1a85025a331: Time spent bootstrapping tablet: real 0.236s user 0.039s sys 0.016s
I20260807 03:39:36.850225 18183 raft_consensus.cc:2932] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Committing config change with OpId 9.8: config changed from index -1 to 8, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:39:36.866106 17918 catalog_manager.cc:5495] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: config changed from index -1 to 8, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New cstate: current_term: 9 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:36.872668 22246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3dc635ee56b4559a87f5955bb83235a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:39:36.888285 22203 raft_consensus.cc:979] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02: : Instructing follower d9716e2bf06442f5b3b0e81087c53c4d to start an election
I20260807 03:39:36.888826 22203 raft_consensus.cc:1067] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 LEADER]: Signalling peer d9716e2bf06442f5b3b0e81087c53c4d to start an election
I20260807 03:39:36.889326 22231 raft_consensus.cc:348] T a21f43f2cbf24aeca08087d9e2fb9b2a P 5c9b5062bdfa46af9396f1a85025a331 [term 6 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:39:36.890202 22231 raft_consensus.cc:729] T a21f43f2cbf24aeca08087d9e2fb9b2a P 5c9b5062bdfa46af9396f1a85025a331 [term 6 LEARNER]: Becoming Follower/Learner. State: Replica: 5c9b5062bdfa46af9396f1a85025a331, State: Initialized, Role: LEARNER
I20260807 03:39:36.890827 22231 consensus_queue.cc:260] T a21f43f2cbf24aeca08087d9e2fb9b2a P 5c9b5062bdfa46af9396f1a85025a331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
W20260807 03:39:37.024190 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 127999us
I20260807 03:39:37.035382 22231 ts_tablet_manager.cc:1428] T a21f43f2cbf24aeca08087d9e2fb9b2a P 5c9b5062bdfa46af9396f1a85025a331: Time spent starting tablet: real 0.184s user 0.016s sys 0.013s
I20260807 03:39:37.062158 18207 tablet_copy_service.cc:342] P 3c4bceb836db4536a0f255fdf86abc02: Request end of tablet copy session 5c9b5062bdfa46af9396f1a85025a331-a21f43f2cbf24aeca08087d9e2fb9b2a received from {username='slave'} at 127.0.0.1:50060
I20260807 03:39:37.062625 18207 tablet_copy_service.cc:433] P 3c4bceb836db4536a0f255fdf86abc02: ending tablet copy session 5c9b5062bdfa46af9396f1a85025a331-a21f43f2cbf24aeca08087d9e2fb9b2a on tablet a21f43f2cbf24aeca08087d9e2fb9b2a with peer 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:39:37.077466 22259 ts_tablet_manager.cc:927] T b15fdfae36e94837adea3c6b66864b0e P 5c9b5062bdfa46af9396f1a85025a331: Initiating tablet copy from peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163)
I20260807 03:39:37.084434 18075 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2"
dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:33268
I20260807 03:39:37.085001 18075 raft_consensus.cc:482] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:39:37.085310 18075 raft_consensus.cc:3037] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:37.085304 19297 raft_consensus.cc:1203] T a21f43f2cbf24aeca08087d9e2fb9b2a P 5c9b5062bdfa46af9396f1a85025a331 [term 6 LEARNER]: Deduplicated request from leader. Original: 6.4->[6.5-6.5] Dedup: 6.5->[]
I20260807 03:39:37.096765 18075 raft_consensus.cc:504] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:37.098650 18075 leader_election.cc:288] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:37.113242 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: true dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:37.113760 18293 raft_consensus.cc:3037] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:37.116237 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: true dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:37.116833 18190 raft_consensus.cc:3032] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 LEADER]: Stepping down as leader of term 8
I20260807 03:39:37.117097 18190 raft_consensus.cc:729] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:37.117200 22183 raft_consensus.cc:979] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49: : Instructing follower 3c4bceb836db4536a0f255fdf86abc02 to start an election
I20260807 03:39:37.117666 22183 raft_consensus.cc:1067] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Signalling peer 3c4bceb836db4536a0f255fdf86abc02 to start an election
I20260807 03:39:37.117779 18190 consensus_queue.cc:260] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:37.118964 18190 raft_consensus.cc:3037] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:37.120654 18293 raft_consensus.cc:2454] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 9.
I20260807 03:39:37.124516 22259 tablet_copy_client.cc:316] T b15fdfae36e94837adea3c6b66864b0e P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Beginning tablet copy session from remote peer at address 127.15.198.129:36163
I20260807 03:39:37.125926 18187 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
from {username='slave'} at 127.0.0.1:44660
I20260807 03:39:37.126087 18094 tablet_copy_service.cc:140] P d9716e2bf06442f5b3b0e81087c53c4d: Received BeginTabletCopySession request for tablet b15fdfae36e94837adea3c6b66864b0e from peer 5c9b5062bdfa46af9396f1a85025a331 ({username='slave'} at 127.0.0.1:58950)
I20260807 03:39:37.126416 18187 raft_consensus.cc:482] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:39:37.126652 18094 tablet_copy_service.cc:161] P d9716e2bf06442f5b3b0e81087c53c4d: Beginning new tablet copy session on tablet b15fdfae36e94837adea3c6b66864b0e from peer 5c9b5062bdfa46af9396f1a85025a331 at {username='slave'} at 127.0.0.1:58950: session id = 5c9b5062bdfa46af9396f1a85025a331-b15fdfae36e94837adea3c6b66864b0e
I20260807 03:39:37.126819 18187 raft_consensus.cc:3037] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:37.140980 22246 tablet_copy_client.cc:791] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Starting download of 0 data blocks...
I20260807 03:39:37.141608 22246 tablet_copy_client.cc:655] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Starting download of 1 WAL segments...
I20260807 03:39:37.147171 18190 raft_consensus.cc:2454] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 9.
I20260807 03:39:37.147959 18094 tablet_copy_source_session.cc:205] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 03:39:37.150035 22246 tablet_copy_client.cc:531] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 03:39:37.153679 22259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b15fdfae36e94837adea3c6b66864b0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:39:37.159188 18187 raft_consensus.cc:504] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:37.166738 22246 tablet_bootstrap.cc:492] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331: Bootstrap starting.
I20260807 03:39:37.171886 18027 leader_election.cc:302] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:37.164383 18187 leader_election.cc:288] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:37.181733 21957 raft_consensus.cc:2781] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader election won for term 9
I20260807 03:39:37.190886 21957 raft_consensus.cc:686] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:37.191843 21957 consensus_queue.cc:237] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:37.196045 22246 tablet_bootstrap.cc:492] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 03:39:37.196964 22246 tablet_bootstrap.cc:492] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331: Bootstrap complete.
I20260807 03:39:37.197691 22246 ts_tablet_manager.cc:1397] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331: Time spent bootstrapping tablet: real 0.031s user 0.028s sys 0.000s
I20260807 03:39:37.200583 22246 raft_consensus.cc:348] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:39:37.201421 22246 raft_consensus.cc:729] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 5c9b5062bdfa46af9396f1a85025a331, State: Initialized, Role: LEARNER
I20260807 03:39:37.202122 22246 consensus_queue.cc:260] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:39:37.205504 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:37.206094 18300 raft_consensus.cc:3032] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:39:37.206375 18300 raft_consensus.cc:729] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:37.207070 22246 ts_tablet_manager.cc:1428] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331: Time spent starting tablet: real 0.009s user 0.003s sys 0.004s
I20260807 03:39:37.207059 18300 consensus_queue.cc:260] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:37.208472 18300 raft_consensus.cc:3037] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:37.214535 18300 raft_consensus.cc:2454] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:39:37.215188 18094 tablet_copy_service.cc:342] P d9716e2bf06442f5b3b0e81087c53c4d: Request end of tablet copy session 5c9b5062bdfa46af9396f1a85025a331-a3dc635ee56b4559a87f5955bb83235a received from {username='slave'} at 127.0.0.1:58950
I20260807 03:39:37.226260 18094 tablet_copy_service.cc:433] P d9716e2bf06442f5b3b0e81087c53c4d: ending tablet copy session 5c9b5062bdfa46af9396f1a85025a331-a3dc635ee56b4559a87f5955bb83235a on tablet a3dc635ee56b4559a87f5955bb83235a with peer 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:39:37.219089 17920 catalog_manager.cc:5495] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 8 to 9, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 9 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:37.225837 22231 ts_tablet_manager.cc:927] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 5c9b5062bdfa46af9396f1a85025a331: Initiating tablet copy from peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163)
I20260807 03:39:37.240388 22259 tablet_copy_client.cc:791] T b15fdfae36e94837adea3c6b66864b0e P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Starting download of 0 data blocks...
I20260807 03:39:37.241055 22259 tablet_copy_client.cc:655] T b15fdfae36e94837adea3c6b66864b0e P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Starting download of 1 WAL segments...
I20260807 03:39:37.241729 18141 leader_election.cc:302] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:37.242486 22210 raft_consensus.cc:2781] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 03:39:37.245910 22132 raft_consensus.cc:979] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d: : Instructing follower 66a36e1ac8e945e893668e282d394c49 to start an election
I20260807 03:39:37.243237 22210 raft_consensus.cc:686] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:37.246393 22132 raft_consensus.cc:1067] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 LEADER]: Signalling peer 66a36e1ac8e945e893668e282d394c49 to start an election
I20260807 03:39:37.247067 22210 consensus_queue.cc:237] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:37.251401 22259 tablet_copy_client.cc:531] T b15fdfae36e94837adea3c6b66864b0e P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 03:39:37.265513 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:37.266264 18080 raft_consensus.cc:3037] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:37.273178 18080 raft_consensus.cc:2454] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:39:37.274595 18293 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88"
dest_uuid: "66a36e1ac8e945e893668e282d394c49"
from {username='slave'} at 127.0.0.1:57276
I20260807 03:39:37.275152 18293 raft_consensus.cc:482] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:39:37.275494 18293 raft_consensus.cc:3037] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:37.277338 19297 raft_consensus.cc:1203] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3] Dedup: 3.3->[]
I20260807 03:39:37.281649 18293 raft_consensus.cc:504] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:37.280472 17920 catalog_manager.cc:5495] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 3 to 4, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 4 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:37.299314 22259 tablet_bootstrap.cc:492] T b15fdfae36e94837adea3c6b66864b0e P 5c9b5062bdfa46af9396f1a85025a331: Bootstrap starting.
I20260807 03:39:37.307454 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: true dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:37.309309 18190 raft_consensus.cc:3037] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:37.309927 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: true dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:37.310536 18080 raft_consensus.cc:3032] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 LEADER]: Stepping down as leader of term 8
I20260807 03:39:37.310801 18080 raft_consensus.cc:729] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:37.311652 18080 consensus_queue.cc:260] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:37.312793 18080 raft_consensus.cc:3037] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:37.318181 18190 raft_consensus.cc:2454] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 9.
I20260807 03:39:37.319300 18249 leader_election.cc:302] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:37.320542 22126 raft_consensus.cc:2781] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader election won for term 9
I20260807 03:39:37.321039 22126 raft_consensus.cc:686] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 9 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:37.321909 22126 consensus_queue.cc:237] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:37.310828 18293 leader_election.cc:288] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:37.343696 22231 tablet_copy_client.cc:316] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Beginning tablet copy session from remote peer at address 127.15.198.129:36163
I20260807 03:39:37.346495 18094 tablet_copy_service.cc:140] P d9716e2bf06442f5b3b0e81087c53c4d: Received BeginTabletCopySession request for tablet d8d26ed9a1e340ce8c2d127ac7fbf7f7 from peer 5c9b5062bdfa46af9396f1a85025a331 ({username='slave'} at 127.0.0.1:58950)
I20260807 03:39:37.346993 18094 tablet_copy_service.cc:161] P d9716e2bf06442f5b3b0e81087c53c4d: Beginning new tablet copy session on tablet d8d26ed9a1e340ce8c2d127ac7fbf7f7 from peer 5c9b5062bdfa46af9396f1a85025a331 at {username='slave'} at 127.0.0.1:58950: session id = 5c9b5062bdfa46af9396f1a85025a331-d8d26ed9a1e340ce8c2d127ac7fbf7f7
I20260807 03:39:37.355365 18094 tablet_copy_source_session.cc:205] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 03:39:37.360669 22231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8d26ed9a1e340ce8c2d127ac7fbf7f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:39:37.355944 17912 catalog_manager.cc:5495] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 8 to 9, leader changed from d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 9 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:39:37.367473 18080 raft_consensus.cc:2454] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 9.
I20260807 03:39:37.409929 22231 tablet_copy_client.cc:791] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Starting download of 0 data blocks...
I20260807 03:39:37.412823 22231 tablet_copy_client.cc:655] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Starting download of 1 WAL segments...
I20260807 03:39:37.421293 22231 tablet_copy_client.cc:531] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 03:39:37.442538 22231 tablet_bootstrap.cc:492] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 5c9b5062bdfa46af9396f1a85025a331: Bootstrap starting.
I20260807 03:39:37.500619 22259 tablet_bootstrap.cc:492] T b15fdfae36e94837adea3c6b66864b0e P 5c9b5062bdfa46af9396f1a85025a331: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 03:39:37.501536 22259 tablet_bootstrap.cc:492] T b15fdfae36e94837adea3c6b66864b0e P 5c9b5062bdfa46af9396f1a85025a331: Bootstrap complete.
I20260807 03:39:37.505017 22259 ts_tablet_manager.cc:1397] T b15fdfae36e94837adea3c6b66864b0e P 5c9b5062bdfa46af9396f1a85025a331: Time spent bootstrapping tablet: real 0.212s user 0.037s sys 0.058s
I20260807 03:39:37.511840 22259 raft_consensus.cc:348] T b15fdfae36e94837adea3c6b66864b0e P 5c9b5062bdfa46af9396f1a85025a331 [term 5 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:39:37.512714 22259 raft_consensus.cc:729] T b15fdfae36e94837adea3c6b66864b0e P 5c9b5062bdfa46af9396f1a85025a331 [term 5 LEARNER]: Becoming Follower/Learner. State: Replica: 5c9b5062bdfa46af9396f1a85025a331, State: Initialized, Role: LEARNER
I20260807 03:39:37.513409 22259 consensus_queue.cc:260] T b15fdfae36e94837adea3c6b66864b0e P 5c9b5062bdfa46af9396f1a85025a331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:39:37.520713 22259 ts_tablet_manager.cc:1428] T b15fdfae36e94837adea3c6b66864b0e P 5c9b5062bdfa46af9396f1a85025a331: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20260807 03:39:37.522809 18094 tablet_copy_service.cc:342] P d9716e2bf06442f5b3b0e81087c53c4d: Request end of tablet copy session 5c9b5062bdfa46af9396f1a85025a331-b15fdfae36e94837adea3c6b66864b0e received from {username='slave'} at 127.0.0.1:58950
I20260807 03:39:37.523216 18094 tablet_copy_service.cc:433] P d9716e2bf06442f5b3b0e81087c53c4d: ending tablet copy session 5c9b5062bdfa46af9396f1a85025a331-b15fdfae36e94837adea3c6b66864b0e on tablet b15fdfae36e94837adea3c6b66864b0e with peer 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:39:37.548877 19297 raft_consensus.cc:1203] T b15fdfae36e94837adea3c6b66864b0e P 5c9b5062bdfa46af9396f1a85025a331 [term 5 LEARNER]: Deduplicated request from leader. Original: 5.4->[5.5-5.5] Dedup: 5.5->[]
I20260807 03:39:37.607280 22257 raft_consensus.cc:1050] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02: attempting to promote NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 to VOTER
I20260807 03:39:37.613080 18190 raft_consensus.cc:1261] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260807 03:39:37.614684 21958 consensus_queue.cc:1035] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 03:39:37.617249 22257 consensus_queue.cc:237] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 3, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:37.694113 18293 raft_consensus.cc:1261] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260807 03:39:37.694850 22231 tablet_bootstrap.cc:492] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 5c9b5062bdfa46af9396f1a85025a331: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 03:39:37.695423 19297 raft_consensus.cc:1261] T a21f43f2cbf24aeca08087d9e2fb9b2a P 5c9b5062bdfa46af9396f1a85025a331 [term 6 LEARNER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260807 03:39:37.701556 22231 tablet_bootstrap.cc:492] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 5c9b5062bdfa46af9396f1a85025a331: Bootstrap complete.
I20260807 03:39:37.703538 22231 ts_tablet_manager.cc:1397] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 5c9b5062bdfa46af9396f1a85025a331: Time spent bootstrapping tablet: real 0.261s user 0.060s sys 0.042s
I20260807 03:39:37.708848 22231 raft_consensus.cc:348] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 5c9b5062bdfa46af9396f1a85025a331 [term 9 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:39:37.710008 22231 raft_consensus.cc:729] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 5c9b5062bdfa46af9396f1a85025a331 [term 9 LEARNER]: Becoming Follower/Learner. State: Replica: 5c9b5062bdfa46af9396f1a85025a331, State: Initialized, Role: LEARNER
I20260807 03:39:37.711069 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:39:37.700316 22210 consensus_queue.cc:1035] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.012s
I20260807 03:39:37.716123 22165 consensus_queue.cc:1035] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20260807 03:39:37.723644 18080 raft_consensus.cc:1261] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260807 03:39:37.710990 22231 consensus_queue.cc:260] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 5c9b5062bdfa46af9396f1a85025a331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:39:37.744477 22210 consensus_queue.cc:1035] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.015s
I20260807 03:39:37.764137 22231 ts_tablet_manager.cc:1428] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 5c9b5062bdfa46af9396f1a85025a331: Time spent starting tablet: real 0.060s user 0.003s sys 0.007s
I20260807 03:39:37.774073 18094 tablet_copy_service.cc:342] P d9716e2bf06442f5b3b0e81087c53c4d: Request end of tablet copy session 5c9b5062bdfa46af9396f1a85025a331-d8d26ed9a1e340ce8c2d127ac7fbf7f7 received from {username='slave'} at 127.0.0.1:58950
I20260807 03:39:37.774540 18094 tablet_copy_service.cc:433] P d9716e2bf06442f5b3b0e81087c53c4d: ending tablet copy session 5c9b5062bdfa46af9396f1a85025a331-d8d26ed9a1e340ce8c2d127ac7fbf7f7 on tablet d8d26ed9a1e340ce8c2d127ac7fbf7f7 with peer 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:39:37.783792 18312 raft_consensus.cc:1261] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260807 03:39:37.794203 21957 consensus_queue.cc:1035] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.007s
I20260807 03:39:37.803745 21957 raft_consensus.cc:1050] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d: attempting to promote NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 to VOTER
I20260807 03:39:37.820060 18293 raft_consensus.cc:1261] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 03:39:37.827589 22164 consensus_queue.cc:1035] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20260807 03:39:37.869876 18075 raft_consensus.cc:1261] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260807 03:39:37.874387 22165 raft_consensus.cc:2932] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Committing config change with OpId 6.6: config changed from index 5 to 6, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:37.880503 22215 consensus_queue.cc:1035] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.014s
W20260807 03:39:37.923424 18246 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.132:41381, user_credentials={real_user=slave}} blocked reactor thread for 46752.8us
I20260807 03:39:37.943296 18190 raft_consensus.cc:1261] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260807 03:39:37.945380 18071 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79"
dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
mode: GRACEFUL
new_leader_uuid: "66a36e1ac8e945e893668e282d394c49"
from {username='slave'} at 127.0.0.1:55742
I20260807 03:39:37.970772 18072 raft_consensus.cc:1261] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 03:39:37.969161 22126 consensus_queue.cc:1035] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20260807 03:39:37.984407 18071 raft_consensus.cc:595] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Received request to transfer leadership to TS 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:37.990022 22257 consensus_queue.cc:1035] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.014s
I20260807 03:39:38.004222 19295 raft_consensus.cc:2932] T a21f43f2cbf24aeca08087d9e2fb9b2a P 5c9b5062bdfa46af9396f1a85025a331 [term 6 FOLLOWER]: Committing config change with OpId 6.6: config changed from index 5 to 6, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:38.024494 21957 consensus_queue.cc:237] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 1, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:38.033015 19297 raft_consensus.cc:1203] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 5c9b5062bdfa46af9396f1a85025a331 [term 9 LEARNER]: Deduplicated request from leader. Original: 9.7->[9.8-9.8] Dedup: 9.8->[]
I20260807 03:39:38.040936 18075 raft_consensus.cc:2932] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Committing config change with OpId 6.6: config changed from index 5 to 6, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
W20260807 03:39:38.147084 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 82088.1us
I20260807 03:39:38.193288 19295 raft_consensus.cc:1261] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331 [term 3 LEARNER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260807 03:39:38.237246 22298 consensus_queue.cc:1035] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 03:39:38.287473 18297 raft_consensus.cc:2932] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Committing config change with OpId 6.6: config changed from index 5 to 6, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:38.299191 22288 raft_consensus.cc:1050] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d: attempting to promote NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 to VOTER
I20260807 03:39:38.308971 18183 raft_consensus.cc:1261] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260807 03:39:38.390800 22288 consensus_queue.cc:237] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 3, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:38.409621 18297 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4ae0265e0707432980672f7cc955c571"
dest_uuid: "66a36e1ac8e945e893668e282d394c49"
mode: GRACEFUL
new_leader_uuid: "3c4bceb836db4536a0f255fdf86abc02"
from {username='slave'} at 127.0.0.1:58736
I20260807 03:39:38.441041 18297 raft_consensus.cc:595] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Received request to transfer leadership to TS 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:38.379088 17914 catalog_manager.cc:5495] T a21f43f2cbf24aeca08087d9e2fb9b2a P 5c9b5062bdfa46af9396f1a85025a331 reported cstate change: config changed from index 5 to 6, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New cstate: current_term: 6 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:38.453871 18293 raft_consensus.cc:1261] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260807 03:39:38.454702 22132 consensus_queue.cc:1035] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 03:39:38.465991 18190 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "60437eed8852484c88e84993e376670e"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
mode: GRACEFUL
new_leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:34758
I20260807 03:39:38.466697 18190 raft_consensus.cc:595] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Received request to transfer leadership to TS d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:38.471186 22298 consensus_queue.cc:1035] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 03:39:38.472456 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260807 03:39:38.478480 22165 raft_consensus.cc:979] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02: : Instructing follower d9716e2bf06442f5b3b0e81087c53c4d to start an election
I20260807 03:39:38.478987 22165 raft_consensus.cc:1067] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Signalling peer d9716e2bf06442f5b3b0e81087c53c4d to start an election
I20260807 03:39:38.480798 22285 raft_consensus.cc:979] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d: : Instructing follower 66a36e1ac8e945e893668e282d394c49 to start an election
I20260807 03:39:38.481525 22285 raft_consensus.cc:1067] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Signalling peer 66a36e1ac8e945e893668e282d394c49 to start an election
I20260807 03:39:38.482110 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:39:38.482970 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:39:38.529909 22132 raft_consensus.cc:1050] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d: attempting to promote NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 to VOTER
I20260807 03:39:38.533514 22132 consensus_queue.cc:237] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 6, Current term: 9, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:38.538754 18069 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "60437eed8852484c88e84993e376670e"
dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:33268
I20260807 03:39:38.539467 18069 raft_consensus.cc:482] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:39:38.539891 18069 raft_consensus.cc:3037] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:38.546149 18300 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79"
dest_uuid: "66a36e1ac8e945e893668e282d394c49"
from {username='slave'} at 127.0.0.1:57276
I20260807 03:39:38.546729 18300 raft_consensus.cc:482] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:39:38.547115 18300 raft_consensus.cc:3037] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:38.550323 18069 raft_consensus.cc:504] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:38.555760 19295 raft_consensus.cc:1261] T b15fdfae36e94837adea3c6b66864b0e P 5c9b5062bdfa46af9396f1a85025a331 [term 5 LEARNER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260807 03:39:38.557194 18069 leader_election.cc:288] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:38.570104 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:38.570899 18310 raft_consensus.cc:3037] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:38.576009 22297 consensus_queue.cc:1035] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.014s
I20260807 03:39:38.576730 18298 raft_consensus.cc:1261] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260807 03:39:38.600775 22183 raft_consensus.cc:979] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49: : Instructing follower 3c4bceb836db4536a0f255fdf86abc02 to start an election
I20260807 03:39:38.601326 22183 raft_consensus.cc:1067] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Signalling peer 3c4bceb836db4536a0f255fdf86abc02 to start an election
I20260807 03:39:38.601326 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:38.602149 18188 raft_consensus.cc:3032] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Stepping down as leader of term 6
I20260807 03:39:38.601451 18187 raft_consensus.cc:1261] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260807 03:39:38.602491 18188 raft_consensus.cc:729] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:38.604959 18187 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4ae0265e0707432980672f7cc955c571"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
from {username='slave'} at 127.0.0.1:44660
I20260807 03:39:38.605723 18187 raft_consensus.cc:482] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:39:38.606055 18187 raft_consensus.cc:3037] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:38.609668 18310 raft_consensus.cc:2454] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 7.
I20260807 03:39:38.612455 18188 consensus_queue.cc:260] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:38.613483 18188 raft_consensus.cc:3037] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:38.622007 22285 consensus_queue.cc:1035] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 03:39:38.622136 18300 raft_consensus.cc:504] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:38.624994 18027 leader_election.cc:302] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:38.625833 22285 raft_consensus.cc:2781] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election won for term 7
I20260807 03:39:38.626405 22285 raft_consensus.cc:686] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 7 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:38.627707 22285 consensus_queue.cc:237] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:38.626268 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:38.633428 18069 raft_consensus.cc:3032] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Stepping down as leader of term 5
I20260807 03:39:38.633807 18069 raft_consensus.cc:729] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:38.634760 18069 consensus_queue.cc:260] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:38.639310 18188 raft_consensus.cc:2454] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 7.
I20260807 03:39:38.640415 18300 leader_election.cc:288] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:38.649739 22291 consensus_queue.cc:1035] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
I20260807 03:39:38.652508 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:38.653232 18183 raft_consensus.cc:3037] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:38.716706 18069 raft_consensus.cc:3037] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:38.705933 18183 raft_consensus.cc:2454] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 6.
I20260807 03:39:38.733907 18249 leader_election.cc:302] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:38.738209 22206 raft_consensus.cc:2781] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 03:39:38.738715 22206 raft_consensus.cc:686] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 6 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:38.739698 22206 consensus_queue.cc:237] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:38.727953 18187 raft_consensus.cc:504] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:38.770826 19295 raft_consensus.cc:1261] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 5c9b5062bdfa46af9396f1a85025a331 [term 9 LEARNER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260807 03:39:38.773279 22298 consensus_queue.cc:1035] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260807 03:39:38.781410 18187 leader_election.cc:288] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:38.793064 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:38.793980 18297 raft_consensus.cc:3032] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Stepping down as leader of term 7
I20260807 03:39:38.794309 18297 raft_consensus.cc:729] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:38.798573 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:38.799214 18075 raft_consensus.cc:3037] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:38.804441 18300 raft_consensus.cc:2932] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:38.808656 18297 consensus_queue.cc:260] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:38.830549 18297 raft_consensus.cc:3037] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:38.864084 18069 raft_consensus.cc:2454] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 6.
I20260807 03:39:38.883009 18309 raft_consensus.cc:1261] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
W20260807 03:39:38.885187 22343 log.cc:924] Time spent T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.075s user 0.000s sys 0.001s
I20260807 03:39:38.895701 18075 raft_consensus.cc:2454] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 8.
I20260807 03:39:38.902232 18297 raft_consensus.cc:2454] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 8.
I20260807 03:39:38.909336 22132 consensus_queue.cc:1035] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.008s
W20260807 03:39:38.919076 22344 log.cc:924] Time spent T 00000000000000000000000000000000 P 8da2065a77024f35bdfc0dc3adb83f69: Append to log took a long time: real 0.051s user 0.001s sys 0.000s
I20260807 03:39:38.927943 18187 raft_consensus.cc:1261] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260807 03:39:38.928143 18140 leader_election.cc:302] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:38.929368 18188 raft_consensus.cc:2932] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:38.930516 22281 raft_consensus.cc:2781] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader election won for term 8
I20260807 03:39:38.931217 22281 raft_consensus.cc:686] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:38.933488 22281 consensus_queue.cc:237] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:38.933168 18297 raft_consensus.cc:2932] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 5 to 6, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:38.942912 22132 consensus_queue.cc:1035] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260807 03:39:38.944659 22285 raft_consensus.cc:2932] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Committing config change with OpId 5.6: config changed from index 5 to 6, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:38.954515 22286 raft_consensus.cc:2932] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:38.950908 19295 raft_consensus.cc:2932] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
W20260807 03:39:38.969544 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 52681.5us
I20260807 03:39:38.988686 19297 raft_consensus.cc:2932] T b15fdfae36e94837adea3c6b66864b0e P 5c9b5062bdfa46af9396f1a85025a331 [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 5 to 6, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:39.060899 22291 raft_consensus.cc:2932] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 LEADER]: Committing config change with OpId 9.9: config changed from index 8 to 9, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
W20260807 03:39:39.119045 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 84007us
W20260807 03:39:39.135802 19239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 54227.3us
W20260807 03:39:39.153612 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 121555us
I20260807 03:39:39.215276 17914 catalog_manager.cc:5495] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331 reported cstate change: config changed from index 3 to 4, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
W20260807 03:39:39.236804 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 116973us
I20260807 03:39:39.239691 18189 raft_consensus.cc:2932] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 5 to 6, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:39.256660 19297 raft_consensus.cc:2932] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 5c9b5062bdfa46af9396f1a85025a331 [term 9 FOLLOWER]: Committing config change with OpId 9.9: config changed from index 8 to 9, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:39.284796 18190 raft_consensus.cc:1261] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260807 03:39:39.284791 18188 consensus_queue.cc:237] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:39.288048 17914 catalog_manager.cc:5495] T b15fdfae36e94837adea3c6b66864b0e P 5c9b5062bdfa46af9396f1a85025a331 reported cstate change: config changed from index 5 to 6, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New cstate: current_term: 5 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:39.296284 18183 raft_consensus.cc:1261] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260807 03:39:39.302984 18310 raft_consensus.cc:1261] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260807 03:39:39.306793 17920 catalog_manager.cc:5495] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 5 to 6, leader changed from d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 6 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:39:39.310015 18075 raft_consensus.cc:1261] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260807 03:39:39.311440 22206 consensus_queue.cc:1035] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 03:39:39.312032 18183 raft_consensus.cc:2932] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Committing config change with OpId 9.9: config changed from index 8 to 9, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:39.310896 17915 catalog_manager.cc:5495] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 5c9b5062bdfa46af9396f1a85025a331 reported cstate change: config changed from index 8 to 9, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New cstate: current_term: 9 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:39.321338 22183 consensus_queue.cc:1035] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 03:39:39.321870 19297 raft_consensus.cc:1261] T a21f43f2cbf24aeca08087d9e2fb9b2a P 5c9b5062bdfa46af9396f1a85025a331 [term 6 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260807 03:39:39.323410 22319 consensus_queue.cc:1035] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260807 03:39:39.331362 22291 consensus_queue.cc:1035] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 03:39:39.342410 18298 raft_consensus.cc:2932] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Committing config change with OpId 9.9: config changed from index 8 to 9, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:39.345424 18297 raft_consensus.cc:1261] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260807 03:39:39.418612 22320 consensus_queue.cc:1035] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 03:39:39.400452 22165 consensus_queue.cc:1035] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.020s
I20260807 03:39:39.468482 18080 raft_consensus.cc:1261] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260807 03:39:39.475921 17912 catalog_manager.cc:5495] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 6 to 7, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 7 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:39.573607 22319 consensus_queue.cc:1035] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.022s
W20260807 03:39:39.618323 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 54744.5us
I20260807 03:39:39.680408 22387 raft_consensus.cc:2932] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Committing config change with OpId 6.7: config changed from index 6 to 7, VOTER d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:39.726521 17899 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet a21f43f2cbf24aeca08087d9e2fb9b2a with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260807 03:39:39.728333 18138 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.132:41381, user_credentials={real_user=slave}} blocked reactor thread for 140794us
W20260807 03:39:39.743873 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 169980us
W20260807 03:39:39.758644 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 150880us
I20260807 03:39:39.939581 19295 raft_consensus.cc:2932] T a21f43f2cbf24aeca08087d9e2fb9b2a P 5c9b5062bdfa46af9396f1a85025a331 [term 6 FOLLOWER]: Committing config change with OpId 6.7: config changed from index 6 to 7, VOTER d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
W20260807 03:39:39.989459 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 201188us
W20260807 03:39:40.132627 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 238162us
W20260807 03:39:40.151242 18025 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.132:41381, user_credentials={real_user=slave}} blocked reactor thread for 130982us
W20260807 03:39:40.239584 18249 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.584s user 0.015s sys 0.009s
W20260807 03:39:40.244860 18249 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.589s user 0.017s sys 0.011s
W20260807 03:39:40.245499 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 590153us
I20260807 03:39:40.213973 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:39:40.274646 17912 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39492 (request call id 214) took 1026 ms. Trace:
I20260807 03:39:40.462846 22367 raft_consensus.cc:482] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:40.488435 22210 raft_consensus.cc:482] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:40.282893 17912 rpcz_store.cc:276] 0807 03:39:39.248253 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:39.249596 (+ 1343us) service_pool.cc:227] Handling call
0807 03:39:40.274575 (+1024979us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 03:39:39.480055 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 03:39:39.560193 (+ 80138us) write_op.cc:432] Acquiring schema lock in shared mode
0807 03:39:39.563372 (+ 3179us) write_op.cc:435] Acquired schema lock
0807 03:39:39.567571 (+ 4199us) tablet.cc:660] Decoding operations
0807 03:39:39.595787 (+ 28216us) write_op.cc:620] Acquiring the partition lock for write op
0807 03:39:39.597420 (+ 1633us) write_op.cc:641] Partition lock acquired for write op
0807 03:39:39.607583 (+ 10163us) tablet.cc:683] Acquiring locks for 1 operations
0807 03:39:39.653718 (+ 46135us) tablet.cc:699] Row locks acquired
0807 03:39:39.656840 (+ 3122us) write_op.cc:260] PREPARE: finished
0807 03:39:39.678235 (+ 21395us) write_op.cc:270] Start()
0807 03:39:39.678346 (+ 111us) write_op.cc:276] Timestamp: P: 1786073979678192 usec, L: 0
0807 03:39:39.678367 (+ 21us) op_driver.cc:331] REPLICATION: starting
0807 03:39:39.678879 (+ 512us) log.cc:842] Serialized 544 byte log entry
0807 03:39:39.680962 (+ 2083us) op_driver.cc:444] REPLICATION: finished
0807 03:39:39.681444 (+ 482us) write_op.cc:301] APPLY: starting
0807 03:39:39.681534 (+ 90us) tablet.cc:1365] starting BulkCheckPresence
0807 03:39:39.681635 (+ 101us) tablet.cc:1368] finished BulkCheckPresence
0807 03:39:39.681754 (+ 119us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 03:39:39.681995 (+ 241us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 03:39:39.682159 (+ 164us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0807 03:39:39.682199 (+ 40us) write_op.cc:312] APPLY: finished
0807 03:39:39.682372 (+ 173us) log.cc:842] Serialized 23 byte log entry
0807 03:39:39.682522 (+ 150us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:39:39.682605 (+ 83us) write_op.cc:454] Released schema lock
0807 03:39:39.682647 (+ 42us) write_op.cc:341] FINISH: Updating metrics
0807 03:39:39.682890 (+ 243us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:39:39.682965 (+ 75us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2245,"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":246,"apply.run_cpu_time_us":1735,"apply.run_wall_time_us":1735,"num_ops":1,"prepare.queue_time_us":98,"prepare.run_cpu_time_us":3131,"prepare.run_wall_time_us":199104,"replication_time_us":2458}]]}
I20260807 03:39:40.593164 22305 raft_consensus.cc:482] T b15fdfae36e94837adea3c6b66864b0e P 5c9b5062bdfa46af9396f1a85025a331 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:40.594023 22305 raft_consensus.cc:504] T b15fdfae36e94837adea3c6b66864b0e P 5c9b5062bdfa46af9396f1a85025a331 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:40.595691 18312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57276 (request call id 1513) took 1113 ms. Trace:
I20260807 03:39:40.595911 22316 raft_consensus.cc:482] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:40.489109 22210 raft_consensus.cc:504] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
W20260807 03:39:40.596913 18069 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.511s user 0.000s sys 0.018s
I20260807 03:39:40.603257 22408 raft_consensus.cc:482] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:40.606988 22412 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:40.463766 22367 raft_consensus.cc:504] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
W20260807 03:39:40.721195 18248 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.023s sys 0.024s
W20260807 03:39:40.871788 19240 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 172831us
I20260807 03:39:40.767256 18310 raft_consensus.cc:1261] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
W20260807 03:39:40.907295 18248 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.733s user 0.030s sys 0.035s
I20260807 03:39:40.611045 18312 rpcz_store.cc:276] 0807 03:39:39.481910 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:39.482025 (+ 115us) service_pool.cc:227] Handling call
0807 03:39:39.482303 (+ 278us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:39.490123 (+ 7820us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:39.490150 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:39:39.564920 (+ 74770us) log.cc:842] Serialized 26 byte log entry
0807 03:39:39.740686 (+175766us) raft_consensus.cc:1631] Marking committed up to 4
0807 03:39:39.740721 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:39.740801 (+ 80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:40.595335 (+854534us) raft_consensus.cc:1671] finished
0807 03:39:40.595358 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:40.595495 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":7751,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":124,"thread_start_us":143271,"threads_started":1,"wal-append.queue_time_us":715114}
W20260807 03:39:40.906772 19239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 193014us
W20260807 03:39:40.780838 18026 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.529s user 0.015s sys 0.019s
W20260807 03:39:40.914871 18026 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.807s user 0.019s sys 0.034s
W20260807 03:39:40.614609 18069 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.529s user 0.000s sys 0.018s
W20260807 03:39:40.722961 18027 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.610s user 0.015s sys 0.039s
W20260807 03:39:40.921875 18027 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.838s user 0.022s sys 0.055s
I20260807 03:39:40.964798 18069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33254 (request call id 2129) took 1091 ms. Trace:
I20260807 03:39:40.884174 22426 raft_consensus.cc:482] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:40.765277 22206 raft_consensus.cc:482] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:40.792498 22418 raft_consensus.cc:482] T d24d0bbea8bd49d482ad95538ca6e631 P 5c9b5062bdfa46af9396f1a85025a331 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:40.790597 22317 raft_consensus.cc:482] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:40.885082 22210 leader_election.cc:288] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:40.965356 18069 rpcz_store.cc:276] 0807 03:39:39.830376 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:39.830522 (+ 146us) service_pool.cc:227] Handling call
0807 03:39:39.830894 (+ 372us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:39.831143 (+ 249us) raft_consensus.cc:1520] Early marking committed up to index 5
0807 03:39:40.085536 (+254393us) log.cc:842] Serialized 14 byte log entry
0807 03:39:40.921005 (+835469us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:40.921053 (+ 48us) raft_consensus.cc:1631] Marking committed up to 5
0807 03:39:40.921101 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:40.921162 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:40.921277 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":440430,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":147748,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":120,"thread_start_us":1083229,"threads_started":2}
W20260807 03:39:40.780534 22291 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.536s user 0.032s sys 0.022s
I20260807 03:39:40.649658 22295 raft_consensus.cc:482] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:40.789776 22411 raft_consensus.cc:482] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:40.981271 19295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:40.931154 22305 leader_election.cc:288] T b15fdfae36e94837adea3c6b66864b0e P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:40.921200 22401 raft_consensus.cc:482] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:40.938591 22303 raft_consensus.cc:482] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:40.924887 22325 raft_consensus.cc:482] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:40.938261 22436 raft_consensus.cc:482] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:40.913038 22316 raft_consensus.cc:504] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:40.987018 22439 raft_consensus.cc:482] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:40.964062 22412 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:40.992525 22440 raft_consensus.cc:482] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:40.976267 22206 raft_consensus.cc:504] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:40.924338 22408 raft_consensus.cc:504] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.005277 22303 raft_consensus.cc:504] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:40.981510 22317 raft_consensus.cc:504] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:40.974756 22426 raft_consensus.cc:504] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.115290 22427 raft_consensus.cc:482] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:40.985512 22411 raft_consensus.cc:504] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:40.978518 22418 raft_consensus.cc:504] T d24d0bbea8bd49d482ad95538ca6e631 P 5c9b5062bdfa46af9396f1a85025a331 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:40.987013 22295 raft_consensus.cc:504] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:40.986105 22401 raft_consensus.cc:504] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.013772 22325 raft_consensus.cc:504] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.144872 22210 raft_consensus.cc:482] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
W20260807 03:39:41.167317 19240 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 53066us
I20260807 03:39:41.047863 22436 raft_consensus.cc:504] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.161026 22291 raft_consensus.cc:482] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:41.064291 22439 raft_consensus.cc:504] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:41.182898 22359 raft_consensus.cc:482] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:41.069746 22440 raft_consensus.cc:504] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:41.194445 22291 raft_consensus.cc:504] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.190961 22359 raft_consensus.cc:504] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:41.195595 22426 leader_election.cc:288] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:41.196412 22465 raft_consensus.cc:482] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:41.196547 22291 leader_election.cc:288] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.196687 22359 leader_election.cc:288] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.197275 22368 raft_consensus.cc:482] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:41.196985 22465 raft_consensus.cc:504] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:41.197779 22368 raft_consensus.cc:504] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.198369 22418 leader_election.cc:288] T d24d0bbea8bd49d482ad95538ca6e631 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:41.199059 22465 leader_election.cc:288] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:41.199307 22412 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:41.199774 22368 leader_election.cc:288] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.200182 22438 raft_consensus.cc:482] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:41.200265 22408 leader_election.cc:288] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.234591 22419 raft_consensus.cc:482] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:41.200899 22447 raft_consensus.cc:482] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:41.201259 22455 raft_consensus.cc:482] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:41.201670 22460 raft_consensus.cc:482] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:41.201980 22432 raft_consensus.cc:482] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:41.202296 22165 raft_consensus.cc:482] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:41.202418 22424 raft_consensus.cc:482] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:41.202826 22456 raft_consensus.cc:482] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:41.203274 22409 raft_consensus.cc:482] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:41.253180 22438 raft_consensus.cc:504] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.265017 22409 raft_consensus.cc:504] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.254767 22447 raft_consensus.cc:504] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.202025 22421 raft_consensus.cc:482] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:41.203728 22401 leader_election.cc:288] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.205017 17920 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39510 (request call id 196) took 1037 ms. Trace:
I20260807 03:39:41.206833 17918 catalog_manager.cc:5495] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 7 to 8, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 8 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:41.208403 18312 raft_consensus.cc:2932] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Committing config change with OpId 6.7: config changed from index 6 to 7, VOTER d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:41.210956 22411 leader_election.cc:288] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.191020 22443 raft_consensus.cc:482] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:41.276684 22421 raft_consensus.cc:504] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.276940 22443 raft_consensus.cc:504] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:41.278640 22421 leader_election.cc:288] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.279090 22443 leader_election.cc:288] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.200563 22446 raft_consensus.cc:482] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:41.279628 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:41.279835 22446 raft_consensus.cc:504] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.280517 22347 raft_consensus.cc:482] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:41.281219 22347 raft_consensus.cc:504] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:41.282012 22446 leader_election.cc:288] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.189214 22390 raft_consensus.cc:482] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:41.223318 22433 raft_consensus.cc:482] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:41.158093 22427 raft_consensus.cc:504] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.264334 22165 raft_consensus.cc:504] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.274849 22424 raft_consensus.cc:504] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.287933 22165 leader_election.cc:288] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.238533 19291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:41.237605 22206 leader_election.cc:288] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163)
I20260807 03:39:41.234652 22316 leader_election.cc:288] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:41.234139 22317 leader_election.cc:288] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.288812 19291 raft_consensus.cc:2454] T d24d0bbea8bd49d482ad95538ca6e631 P 5c9b5062bdfa46af9396f1a85025a331 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:39:41.226971 22295 leader_election.cc:288] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.230473 22303 leader_election.cc:288] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:41.232167 22325 leader_election.cc:288] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:41.236693 22440 leader_election.cc:288] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:41.213866 19297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:41.236584 22441 raft_consensus.cc:482] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:41.289888 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:41.227485 18072 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33268 (request call id 1790) took 1009 ms. Trace:
I20260807 03:39:41.290273 22441 raft_consensus.cc:504] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:41.180751 22210 raft_consensus.cc:504] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.290781 22473 raft_consensus.cc:482] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:41.282717 22390 raft_consensus.cc:504] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
W20260807 03:39:41.291217 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 59412.7us
I20260807 03:39:41.291574 22473 raft_consensus.cc:504] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.287981 22427 leader_election.cc:288] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:41.275365 17920 rpcz_store.cc:276] 0807 03:39:40.167156 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:40.167321 (+ 165us) service_pool.cc:227] Handling call
0807 03:39:41.204956 (+1037635us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":77808,"reported_tablets":3}
I20260807 03:39:41.283778 22433 raft_consensus.cc:504] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.290653 18072 rpcz_store.cc:276] 0807 03:39:40.183679 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:40.186063 (+ 2384us) service_pool.cc:227] Handling call
0807 03:39:40.186527 (+ 464us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:40.709133 (+522606us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:40.709183 (+ 50us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:39:40.902305 (+193122us) log.cc:842] Serialized 26 byte log entry
0807 03:39:41.116948 (+214643us) raft_consensus.cc:1631] Marking committed up to 6
0807 03:39:41.116994 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:41.117079 (+ 85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:41.193325 (+ 76246us) raft_consensus.cc:1671] finished
0807 03:39:41.193342 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:41.193453 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31096,"raft.queue_time_us":54803,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":123,"spinlock_wait_cycles":13696,"thread_start_us":611708,"threads_started":2,"wal-append.queue_time_us":151439}
I20260807 03:39:41.292233 18191 raft_consensus.cc:2454] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 5.
I20260807 03:39:41.302381 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:41.269625 22367 leader_election.cc:288] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:41.309254 18138 leader_election.cc:302] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331; no voters:
W20260807 03:39:41.326370 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 171451us
I20260807 03:39:41.326308 22481 raft_consensus.cc:482] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:41.302611 19290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:41.327145 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:41.330016 18185 raft_consensus.cc:2454] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 3.
I20260807 03:39:41.332484 22436 leader_election.cc:288] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.335239 22473 leader_election.cc:288] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.277367 22463 raft_consensus.cc:482] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:41.256228 22419 raft_consensus.cc:504] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:41.310235 22409 leader_election.cc:288] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.335947 22463 raft_consensus.cc:504] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:41.337128 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:41.338239 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:41.338855 22463 leader_election.cc:288] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:41.339243 22419 leader_election.cc:288] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:41.339388 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:41.340534 22441 leader_election.cc:288] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:41.258106 22460 raft_consensus.cc:504] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.340941 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ca0223f47b46fd955704eba5259370" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:41.341990 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:41.342963 22460 leader_election.cc:288] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.269259 22455 raft_consensus.cc:504] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.344662 22438 leader_election.cc:288] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.342880 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:41.347862 18300 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4ae0265e0707432980672f7cc955c571"
dest_uuid: "66a36e1ac8e945e893668e282d394c49"
mode: GRACEFUL
new_leader_uuid: "3c4bceb836db4536a0f255fdf86abc02"
from {username='slave'} at 127.0.0.1:58736
I20260807 03:39:41.348582 18300 raft_consensus.cc:595] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Received request to transfer leadership to TS 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:41.348961 18300 raft_consensus.cc:603] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260807 03:39:41.356135 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 55085.2us
I20260807 03:39:41.304253 22470 raft_consensus.cc:482] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
W20260807 03:39:41.360984 18138 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.132:41381, user_credentials={real_user=slave}} blocked reactor thread for 51799.9us
I20260807 03:39:41.365574 22279 raft_consensus.cc:2781] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:41.366189 22279 raft_consensus.cc:482] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:41.366520 22349 raft_consensus.cc:482] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:41.366690 22279 raft_consensus.cc:3037] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:41.367089 22349 raft_consensus.cc:504] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.284608 22456 raft_consensus.cc:504] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.368818 22472 raft_consensus.cc:482] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:41.369432 22472 raft_consensus.cc:504] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:41.359193 22436 raft_consensus.cc:482] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:41.322798 22427 raft_consensus.cc:482] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:41.370977 22436 raft_consensus.cc:504] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.371518 22427 raft_consensus.cc:504] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.373400 22447 leader_election.cc:288] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.311746 22424 leader_election.cc:288] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.374346 22439 leader_election.cc:288] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.355856 22206 raft_consensus.cc:482] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:41.375811 22206 raft_consensus.cc:504] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.367610 18079 consensus_queue.cc:237] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:41.261132 22432 raft_consensus.cc:504] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.301908 22390 leader_election.cc:288] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:41.381389 22303 raft_consensus.cc:482] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:41.381938 22303 raft_consensus.cc:504] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.392751 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:41.399749 19297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:41.395176 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:41.400884 19240 leader_election.cc:302] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331; no voters:
I20260807 03:39:41.404438 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ca0223f47b46fd955704eba5259370" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:41.403672 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 76343.1us
W20260807 03:39:41.408645 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 62341us
I20260807 03:39:41.362421 22481 raft_consensus.cc:504] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.412995 19292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:41.413113 19290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:41.413772 19290 raft_consensus.cc:2454] T b15fdfae36e94837adea3c6b66864b0e P 5c9b5062bdfa46af9396f1a85025a331 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 5.
I20260807 03:39:41.414052 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:41.414363 22481 leader_election.cc:288] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.414583 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:41.399019 22473 raft_consensus.cc:482] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 5c9b5062bdfa46af9396f1a85025a331)
I20260807 03:39:41.415292 22473 raft_consensus.cc:504] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:41.415618 22419 consensus_queue.cc:1035] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260807 03:39:41.417311 22473 leader_election.cc:288] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:41.418157 22432 leader_election.cc:288] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.418684 22401 raft_consensus.cc:482] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:41.419387 22411 raft_consensus.cc:482] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:41.419276 22401 raft_consensus.cc:504] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:41.420971 22460 raft_consensus.cc:482] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:41.444155 22460 raft_consensus.cc:504] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.444810 18073 consensus_queue.cc:237] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:41.420511 22475 raft_consensus.cc:482] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:41.447115 22475 raft_consensus.cc:504] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.410054 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:41.463079 22347 leader_election.cc:288] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
W20260807 03:39:41.464078 22484 log.cc:924] Time spent T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.137s user 0.001s sys 0.000s
I20260807 03:39:41.448166 22206 leader_election.cc:288] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:41.503674 19292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:41.465066 22439 raft_consensus.cc:2781] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:41.468518 22460 leader_election.cc:288] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.468604 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:41.394655 22470 raft_consensus.cc:504] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.420248 22317 raft_consensus.cc:482] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:41.449571 18293 raft_consensus.cc:2454] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 6.
I20260807 03:39:41.420778 22424 raft_consensus.cc:482] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
W20260807 03:39:41.496609 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 87140.3us
W20260807 03:39:41.496806 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 148952us
I20260807 03:39:41.504292 18309 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01"
dest_uuid: "66a36e1ac8e945e893668e282d394c49"
mode: GRACEFUL
new_leader_uuid: "3c4bceb836db4536a0f255fdf86abc02"
from {username='slave'} at 127.0.0.1:58736
I20260807 03:39:41.504609 19292 raft_consensus.cc:2454] T b15fdfae36e94837adea3c6b66864b0e P 5c9b5062bdfa46af9396f1a85025a331 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:41.419952 19290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:41.420444 22285 raft_consensus.cc:482] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:41.531991 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:41.525079 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:41.529188 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:41.526053 18309 raft_consensus.cc:595] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Received request to transfer leadership to TS 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:41.530256 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:41.530373 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:41.531363 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:41.496459 22481 raft_consensus.cc:482] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:41.536273 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:41.511451 22439 raft_consensus.cc:482] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:41.517984 22210 leader_election.cc:288] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.420011 22411 raft_consensus.cc:504] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.521138 22432 raft_consensus.cc:482] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:41.531234 22349 leader_election.cc:288] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:41.601176 22449 raft_consensus.cc:482] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:41.585721 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:41.572582 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:41.762475 22449 raft_consensus.cc:504] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.573319 22288 raft_consensus.cc:482] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:41.763404 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:41.680213 22279 raft_consensus.cc:504] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:41.527143 22446 raft_consensus.cc:482] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:41.764892 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:41.765332 22446 raft_consensus.cc:504] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.766093 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:41.766273 18188 raft_consensus.cc:2454] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 7.
I20260807 03:39:41.766395 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:41.767078 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:41.767283 22446 leader_election.cc:288] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.767664 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:41.690167 19290 raft_consensus.cc:1261] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331 [term 3 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260807 03:39:41.572983 22285 raft_consensus.cc:504] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.609275 22470 leader_election.cc:288] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:41.622568 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:41.773077 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:41.691624 19295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:41.541287 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:41.764263 22288 raft_consensus.cc:504] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.768925 19297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331"
I20260807 03:39:41.774412 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:41.774649 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:41.749953 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:41.775822 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:41.692001 21958 raft_consensus.cc:482] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:41.776728 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:41.676155 22475 leader_election.cc:288] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.777108 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:41.777756 22475 consensus_queue.cc:1035] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 03:39:41.778426 22279 leader_election.cc:288] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:41.540668 22424 raft_consensus.cc:504] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:41.782076 22424 leader_election.cc:288] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:41.610239 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:41.622203 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:41.537859 22303 leader_election.cc:288] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:41.551920 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:41.770149 22432 raft_consensus.cc:504] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.784154 19291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:41.785734 22432 leader_election.cc:288] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.531512 22317 raft_consensus.cc:504] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.545436 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:41.573390 22439 raft_consensus.cc:3037] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:41.788641 19240 leader_election.cc:302] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:41.789813 22443 raft_consensus.cc:2726] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 03:39:41.789981 22317 leader_election.cc:288] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.573275 22455 leader_election.cc:288] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.595387 22427 leader_election.cc:288] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:41.538605 22433 leader_election.cc:288] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.525087 22291 raft_consensus.cc:482] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:41.640848 18141 leader_election.cc:302] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49
W20260807 03:39:41.798408 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 157646us
I20260807 03:39:41.676678 22401 leader_election.cc:288] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:41.527070 22436 leader_election.cc:288] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.608194 18300 raft_consensus.cc:2454] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 5.
I20260807 03:39:41.606530 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:41.680146 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:41.768925 22460 raft_consensus.cc:482] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:41.801815 22446 raft_consensus.cc:2726] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:41.802040 18075 raft_consensus.cc:2454] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 1.
I20260807 03:39:41.802091 22460 raft_consensus.cc:504] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.609692 19292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:41.777954 19297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:41.804265 22460 leader_election.cc:288] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.720758 22298 raft_consensus.cc:482] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:41.805684 22298 raft_consensus.cc:504] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.807628 22411 leader_election.cc:288] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.775847 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:41.809974 22298 leader_election.cc:288] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.777119 21958 raft_consensus.cc:504] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.689293 22368 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:41.629487 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:41.728989 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:41.811795 22368 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.630836 22472 leader_election.cc:288] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:41.595489 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:41.541014 22481 raft_consensus.cc:504] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.814143 22368 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.629586 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:41.527298 22456 leader_election.cc:288] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.812731 21958 leader_election.cc:288] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.742748 18309 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf"
dest_uuid: "66a36e1ac8e945e893668e282d394c49"
mode: GRACEFUL
new_leader_uuid: "3c4bceb836db4536a0f255fdf86abc02"
from {username='slave'} at 127.0.0.1:58736
I20260807 03:39:41.807109 22473 raft_consensus.cc:482] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:41.815346 22481 leader_election.cc:288] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.815591 18309 raft_consensus.cc:595] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Received request to transfer leadership to TS 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:41.815863 18082 raft_consensus.cc:2454] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:39:41.815722 22473 raft_consensus.cc:504] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.777120 22288 leader_election.cc:288] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.695926 22462 raft_consensus.cc:482] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:41.818506 22473 leader_election.cc:288] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.818405 19239 leader_election.cc:302] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:41.807343 19240 leader_election.cc:302] T b15fdfae36e94837adea3c6b66864b0e P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:41.819006 22462 raft_consensus.cc:504] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.820117 22443 raft_consensus.cc:2781] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:39:41.820523 22443 raft_consensus.cc:482] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:41.820894 22443 raft_consensus.cc:3037] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:41.821197 22462 leader_election.cc:288] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.797559 22291 raft_consensus.cc:504] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.823163 22291 leader_election.cc:288] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.823742 19292 raft_consensus.cc:1261] T b15fdfae36e94837adea3c6b66864b0e P 5c9b5062bdfa46af9396f1a85025a331 [term 5 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260807 03:39:41.541040 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:41.825552 18070 raft_consensus.cc:2454] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:39:41.806661 22449 leader_election.cc:288] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:41.831884 22462 consensus_queue.cc:1035] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260807 03:39:41.837042 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 103283us
I20260807 03:39:41.837670 18027 leader_election.cc:302] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:41.840791 22288 raft_consensus.cc:2781] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:39:41.841248 22288 raft_consensus.cc:482] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:41.841622 22288 raft_consensus.cc:3037] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:41.847060 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:41.847100 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:41.847313 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:41.848758 22481 raft_consensus.cc:2932] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:41.858098 22288 raft_consensus.cc:504] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.859347 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:41.859926 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:41.861220 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:41.861747 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:41.864706 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:41.862390 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:41.867763 17898 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet a3dc635ee56b4559a87f5955bb83235a with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 03:39:41.867049 19292 raft_consensus.cc:2932] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:41.850222 22443 raft_consensus.cc:504] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:41.878001 22443 leader_election.cc:288] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 2 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.879079 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:41.879624 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:41.880748 22446 raft_consensus.cc:482] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:41.881261 22446 raft_consensus.cc:504] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:41.775729 22285 leader_election.cc:288] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.860021 22288 leader_election.cc:288] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.874941 18141 leader_election.cc:302] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:41.883235 22446 leader_election.cc:288] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:41.883322 22456 raft_consensus.cc:2726] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:41.884565 18141 leader_election.cc:302] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:41.885326 18027 leader_election.cc:302] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:41.886140 22291 raft_consensus.cc:2781] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:39:41.886571 22291 raft_consensus.cc:482] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:41.886919 18027 leader_election.cc:302] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:41.887018 22291 raft_consensus.cc:3037] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:41.888675 22288 raft_consensus.cc:2726] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260807 03:39:41.889317 18249 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.503s user 0.011s sys 0.032s
W20260807 03:39:41.889632 18249 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.503s user 0.011s sys 0.032s
I20260807 03:39:41.889683 22456 raft_consensus.cc:2726] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260807 03:39:41.889976 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 503673us
I20260807 03:39:41.891024 18141 leader_election.cc:302] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:41.892907 22456 raft_consensus.cc:2726] T 9a59a83a64df475ea15dd572f84024bd P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 03:39:41.888000 18027 leader_election.cc:302] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:41.898339 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:41.898628 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:41.898895 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:41.900022 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:41.900604 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:41.901058 22401 raft_consensus.cc:2726] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:41.901968 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:41.825626 22505 raft_consensus.cc:2726] T b15fdfae36e94837adea3c6b66864b0e P 5c9b5062bdfa46af9396f1a85025a331 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:41.904783 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:41.905566 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:41.906049 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:41.908074 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:41.908389 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:41.905591 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:41.910719 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:41.911042 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:41.914074 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:41.896601 18027 leader_election.cc:302] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:41.917001 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:41.917675 18027 leader_election.cc:302] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:41.918948 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:41.921262 22401 raft_consensus.cc:2726] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260807 03:39:41.973142 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 43572.2us
I20260807 03:39:41.976833 22449 raft_consensus.cc:482] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:41.978055 22472 raft_consensus.cc:482] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:41.978518 22303 raft_consensus.cc:482] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
W20260807 03:39:41.978595 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 50118.9us
I20260807 03:39:41.981071 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:41.990159 22439 raft_consensus.cc:504] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:41.996083 18140 leader_election.cc:302] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:42.014097 18140 leader_election.cc:302] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:41.920696 22288 raft_consensus.cc:2726] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260807 03:39:42.012425 22449 raft_consensus.cc:504] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:42.018059 18140 leader_election.cc:302] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:42.019155 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:42.016695 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:42.030164 22303 raft_consensus.cc:504] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:42.017343 22472 raft_consensus.cc:504] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:42.032145 22472 leader_election.cc:288] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:42.032226 22303 leader_election.cc:288] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:42.024143 22210 raft_consensus.cc:2726] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:42.034654 22449 leader_election.cc:288] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:42.035208 22446 raft_consensus.cc:2726] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:41.983726 22291 raft_consensus.cc:504] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:42.037390 18140 leader_election.cc:302] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:42.038040 22291 leader_election.cc:288] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:42.038808 18140 leader_election.cc:302] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:42.039744 22409 raft_consensus.cc:2726] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260807 03:39:42.041378 22409 raft_consensus.cc:2781] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:42.041553 22279 raft_consensus.cc:2726] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260807 03:39:42.041726 22409 raft_consensus.cc:482] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:42.042160 22409 raft_consensus.cc:3037] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:42.045060 19295 raft_consensus.cc:1226] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 4 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:42.045832 22439 leader_election.cc:288] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 4 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:42.047377 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:42.047974 18298 raft_consensus.cc:2454] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:39:42.025048 18249 leader_election.cc:302] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:39:42.051416 22472 raft_consensus.cc:2726] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 03:39:42.052386 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:42.052497 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:42.056511 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:42.052816 18249 leader_election.cc:302] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:39:42.061407 18140 leader_election.cc:302] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:42.066639 18140 leader_election.cc:302] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:42.073912 18141 leader_election.cc:302] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:42.073585 22303 raft_consensus.cc:2726] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260807 03:39:42.075201 22371 raft_consensus.cc:2781] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:42.077768 22371 raft_consensus.cc:482] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:42.078167 22371 raft_consensus.cc:3037] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:42.080245 22210 raft_consensus.cc:2726] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:42.082074 22446 raft_consensus.cc:2781] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:42.082473 22446 raft_consensus.cc:482] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:42.082799 22446 raft_consensus.cc:3037] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:42.090870 22446 raft_consensus.cc:504] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:42.096381 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:42.096517 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:42.097350 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:42.096412 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:42.098169 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:42.100049 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:42.100116 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:42.102360 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 102324us
I20260807 03:39:42.100572 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:42.103396 18073 raft_consensus.cc:2454] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:39:42.103683 18027 leader_election.cc:302] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:42.104262 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:42.101876 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:42.102464 18075 raft_consensus.cc:2454] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 6.
I20260807 03:39:42.105396 18027 leader_election.cc:302] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:42.105631 19240 leader_election.cc:302] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:42.106240 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:42.106819 18075 raft_consensus.cc:3037] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:42.106863 22505 raft_consensus.cc:2726] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 03:39:42.109939 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:42.110531 18081 raft_consensus.cc:2454] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 5.
I20260807 03:39:42.111303 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:42.112247 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:42.113303 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:42.114383 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:42.115100 18073 raft_consensus.cc:2454] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:42.116513 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:42.117115 18079 raft_consensus.cc:2454] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 11.
I20260807 03:39:42.117942 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:42.118481 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:42.115818 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:42.120702 18075 raft_consensus.cc:2454] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 2.
I20260807 03:39:42.123835 18081 raft_consensus.cc:1226] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 1. Current term is 2. Ops: []
I20260807 03:39:42.128453 22409 raft_consensus.cc:504] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:42.130549 22409 leader_election.cc:288] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:42.131469 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:42.132818 22132 raft_consensus.cc:482] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:42.105538 18249 leader_election.cc:302] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:39:42.133450 22132 raft_consensus.cc:504] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:42.134025 22472 raft_consensus.cc:2726] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:42.135686 22285 raft_consensus.cc:2726] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:42.136991 22449 raft_consensus.cc:979] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49: : Instructing follower 3c4bceb836db4536a0f255fdf86abc02 to start an election
I20260807 03:39:42.137480 22303 raft_consensus.cc:1067] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Signalling peer 3c4bceb836db4536a0f255fdf86abc02 to start an election
I20260807 03:39:42.146315 21958 raft_consensus.cc:2726] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260807 03:39:42.147231 19239 leader_election.cc:302] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:42.148324 22439 raft_consensus.cc:2781] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:39:42.149112 22439 raft_consensus.cc:686] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 2 LEADER]: Becoming Leader. State: Replica: 5c9b5062bdfa46af9396f1a85025a331, State: Running, Role: LEADER
I20260807 03:39:42.150125 22439 consensus_queue.cc:237] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:42.144590 18071 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4"
dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
mode: GRACEFUL
new_leader_uuid: "66a36e1ac8e945e893668e282d394c49"
from {username='slave'} at 127.0.0.1:55742
W20260807 03:39:42.153264 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 47956us
I20260807 03:39:42.153942 18027 leader_election.cc:302] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:42.156989 18027 leader_election.cc:302] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:42.162938 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:42.164577 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:42.164813 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:42.152954 18071 raft_consensus.cc:595] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 LEADER]: Received request to transfer leadership to TS 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:42.165462 18188 raft_consensus.cc:2454] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 6.
I20260807 03:39:42.176872 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:42.171731 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:42.176055 19239 leader_election.cc:302] T d24d0bbea8bd49d482ad95538ca6e631 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:42.179987 18027 leader_election.cc:302] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:42.180720 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:42.181394 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:42.183516 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:42.184147 18310 raft_consensus.cc:3037] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:42.185808 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:42.201740 18080 raft_consensus.cc:1226] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:42.168076 21958 raft_consensus.cc:2726] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 03:39:42.203903 18027 leader_election.cc:302] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:42.202733 22443 raft_consensus.cc:2726] T d24d0bbea8bd49d482ad95538ca6e631 P 5c9b5062bdfa46af9396f1a85025a331 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:42.209362 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:42.206624 18069 raft_consensus.cc:1226] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 6. Current term is 7. Ops: []
I20260807 03:39:42.210004 18027 leader_election.cc:302] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:42.209668 18249 leader_election.cc:302] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:42.212245 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:42.181324 22288 raft_consensus.cc:2726] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 03:39:42.222651 18189 raft_consensus.cc:1226] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:42.215137 22470 raft_consensus.cc:2781] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:39:42.181564 22303 consensus_queue.cc:1046] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20260807 03:39:42.215529 22367 raft_consensus.cc:482] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:42.213407 22298 raft_consensus.cc:2726] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260807 03:39:42.231998 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:42.215595 18180 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
from {username='slave'} at 127.0.0.1:44660
I20260807 03:39:42.233011 18180 raft_consensus.cc:482] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:39:42.233515 18180 raft_consensus.cc:3037] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:42.217386 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:42.219440 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:42.235193 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:42.213397 22411 raft_consensus.cc:2726] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:42.182183 22132 leader_election.cc:288] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:42.200914 22462 raft_consensus.cc:2781] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 03:39:42.238937 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:42.239795 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:42.224452 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:42.242295 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:42.225383 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:42.230155 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:42.231364 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:42.244884 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:42.246467 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:42.247169 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:42.248298 18188 raft_consensus.cc:1261] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260807 03:39:42.251091 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:42.251396 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:42.186252 22446 leader_election.cc:288] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:42.237884 18311 raft_consensus.cc:1261] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260807 03:39:42.242995 18310 raft_consensus.cc:2454] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:39:42.252658 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:42.255287 18300 raft_consensus.cc:1226] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:42.227082 22470 raft_consensus.cc:482] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:42.253736 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:42.251816 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:42.240013 22462 raft_consensus.cc:482] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:42.235194 22426 raft_consensus.cc:3032] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:39:42.236114 22367 raft_consensus.cc:504] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:42.264683 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:42.266041 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
W20260807 03:39:42.266454 22515 log.cc:924] Time spent T b15fdfae36e94837adea3c6b66864b0e P 5c9b5062bdfa46af9396f1a85025a331: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20260807 03:39:42.236958 22371 raft_consensus.cc:504] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:42.302094 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:42.269008 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:42.293103 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:42.274453 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:42.668135 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:42.661088 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:42.677086 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:42.678876 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:42.674263 18180 raft_consensus.cc:504] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:42.262327 22470 raft_consensus.cc:3037] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:42.686100 18188 raft_consensus.cc:1226] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:42.688612 18180 leader_election.cc:288] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:42.662971 22426 raft_consensus.cc:729] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:42.667475 22462 raft_consensus.cc:3037] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:42.705817 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:42.706427 18193 raft_consensus.cc:2454] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 3.
I20260807 03:39:42.719220 18190 raft_consensus.cc:1226] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 4. Current term is 5. Ops: []
I20260807 03:39:42.715662 22371 leader_election.cc:288] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:42.729584 22285 raft_consensus.cc:2932] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Committing config change with OpId 5.7: config changed from index 6 to 7, VOTER 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:42.732923 22470 raft_consensus.cc:504] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:42.739202 19290 raft_consensus.cc:2932] T b15fdfae36e94837adea3c6b66864b0e P 5c9b5062bdfa46af9396f1a85025a331 [term 5 FOLLOWER]: Committing config change with OpId 5.7: config changed from index 6 to 7, VOTER 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:42.735769 22470 leader_election.cc:288] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:42.718312 22426 consensus_queue.cc:260] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:42.748813 18070 raft_consensus.cc:1261] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Refusing update from remote peer 5c9b5062bdfa46af9396f1a85025a331: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 03:39:42.750741 22443 consensus_queue.cc:1035] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 03:39:42.761139 22367 leader_election.cc:288] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:42.770015 18027 leader_election.cc:302] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:39:42.772382 22298 consensus_queue.cc:1046] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20260807 03:39:42.772218 22132 raft_consensus.cc:2726] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 03:39:42.780999 22298 raft_consensus.cc:3032] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:39:42.781544 22298 raft_consensus.cc:729] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:42.782577 22298 consensus_queue.cc:260] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:42.786110 22298 raft_consensus.cc:3037] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:42.755800 22462 raft_consensus.cc:504] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:42.789908 22462 leader_election.cc:288] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:42.774447 22401 consensus_queue.cc:1035] T b15fdfae36e94837adea3c6b66864b0e P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.011s
I20260807 03:39:42.791445 18027 leader_election.cc:302] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:42.793926 22401 raft_consensus.cc:2781] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:42.794468 22401 raft_consensus.cc:482] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:42.794992 22401 raft_consensus.cc:3037] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:42.854399 18027 leader_election.cc:302] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:42.855191 21958 consensus_queue.cc:1046] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260807 03:39:42.856269 21958 raft_consensus.cc:3032] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 LEADER]: Stepping down as leader of term 4
I20260807 03:39:42.856608 21958 raft_consensus.cc:729] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:42.857136 22401 raft_consensus.cc:504] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:42.858528 22481 raft_consensus.cc:2726] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:42.860673 18027 leader_election.cc:302] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:42.862015 18027 leader_election.cc:302] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:42.857342 21958 consensus_queue.cc:260] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:42.862406 22132 raft_consensus.cc:2726] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:42.863998 21958 raft_consensus.cc:3037] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:42.868244 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:42.887678 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:42.888466 18190 raft_consensus.cc:3037] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:42.881080 22401 leader_election.cc:288] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:42.902063 18193 raft_consensus.cc:2932] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Committing config change with OpId 5.7: config changed from index 6 to 7, VOTER 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:42.916977 22472 raft_consensus.cc:482] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:42.917291 18312 raft_consensus.cc:3032] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 03:39:42.917800 18312 raft_consensus.cc:729] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:42.917544 22472 raft_consensus.cc:504] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:42.918540 18312 consensus_queue.cc:260] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:42.919541 18312 raft_consensus.cc:3037] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:39:42.919638 22472 leader_election.cc:288] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:42.920516 18180 raft_consensus.cc:1226] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 7. Current term is 8. Ops: []
I20260807 03:39:42.920461 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:42.932427 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:42.933094 18183 raft_consensus.cc:1226] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 4. Current term is 5. Ops: []
I20260807 03:39:42.936030 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:42.937180 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:42.937697 22303 consensus_queue.cc:1046] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 03:39:42.938997 22427 raft_consensus.cc:3032] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Stepping down as leader of term 7
I20260807 03:39:42.938769 22303 consensus_queue.cc:1046] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 03:39:42.939401 22427 raft_consensus.cc:729] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:42.940064 22303 raft_consensus.cc:3032] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:39:42.940907 22303 raft_consensus.cc:729] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:42.941856 22303 consensus_queue.cc:260] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:42.942144 18140 leader_election.cc:302] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:42.942960 22303 raft_consensus.cc:3037] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:42.943544 18140 leader_election.cc:302] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:42.944665 18140 leader_election.cc:302] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:42.945581 22367 raft_consensus.cc:2726] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:42.946132 22367 raft_consensus.cc:2726] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:42.946673 22367 raft_consensus.cc:2781] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:42.947000 22367 raft_consensus.cc:482] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:42.947351 22367 raft_consensus.cc:3037] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:42.951562 22426 raft_consensus.cc:3037] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:42.952535 18248 leader_election.cc:302] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:42.953601 18248 leader_election.cc:302] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:42.954644 18248 leader_election.cc:302] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:42.957119 18140 leader_election.cc:302] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:42.960485 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:42.961122 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:42.962668 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:42.964830 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:42.966943 18027 leader_election.cc:302] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:42.968070 18027 leader_election.cc:302] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:42.969450 18027 leader_election.cc:302] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:42.954443 19290 raft_consensus.cc:1226] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 2 LEADER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 1. Current term is 2. Ops: []
I20260807 03:39:42.971498 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:42.977182 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:42.977823 18189 raft_consensus.cc:2379] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 5: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:42.972335 18140 leader_election.cc:302] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:42.972637 18027 leader_election.cc:302] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:42.972687 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:42.973498 18248 leader_election.cc:302] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:42.978915 18140 leader_election.cc:302] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:42.981125 18140 leader_election.cc:302] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:42.980473 22481 consensus_queue.cc:1035] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20260807 03:39:42.982920 17915 catalog_manager.cc:5495] T a21f43f2cbf24aeca08087d9e2fb9b2a P 5c9b5062bdfa46af9396f1a85025a331 reported cstate change: config changed from index 6 to 7, VOTER d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) evicted. New cstate: current_term: 6 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:42.985378 22479 consensus_queue.cc:1046] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:39:42.986398 22473 raft_consensus.cc:2726] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 03:39:42.971724 22446 raft_consensus.cc:2781] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:39:42.988353 22446 raft_consensus.cc:482] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:42.988562 22531 raft_consensus.cc:2726] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 03:39:42.988744 22446 raft_consensus.cc:3037] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:42.971467 22411 raft_consensus.cc:2726] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:39:42.980515 22371 raft_consensus.cc:2726] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 03:39:42.981032 22449 raft_consensus.cc:2726] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:42.976068 22401 raft_consensus.cc:2726] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:42.980890 22472 raft_consensus.cc:2726] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 03:39:42.980764 22470 raft_consensus.cc:2726] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:42.995632 18187 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
mode: GRACEFUL
new_leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:34758
I20260807 03:39:42.996311 18187 raft_consensus.cc:595] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 LEADER]: Received request to transfer leadership to TS d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:43.007309 17918 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39494 (request call id 191) took 3735 ms. Trace:
I20260807 03:39:43.007546 22479 raft_consensus.cc:2726] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:43.008174 22479 raft_consensus.cc:3032] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 03:39:43.008636 22479 raft_consensus.cc:729] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:43.010355 18248 leader_election.cc:302] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:43.010982 18189 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
mode: GRACEFUL
new_leader_uuid: "5c9b5062bdfa46af9396f1a85025a331"
from {username='slave'} at 127.0.0.1:34758
I20260807 03:39:43.012620 22479 consensus_queue.cc:260] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:43.014483 22479 raft_consensus.cc:3037] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:43.013736 22519 raft_consensus.cc:2781] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:39:43.019188 22519 raft_consensus.cc:482] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:43.019042 22446 raft_consensus.cc:504] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:43.019706 22519 raft_consensus.cc:3037] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:42.974967 21958 raft_consensus.cc:2726] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260807 03:39:43.034519 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:43.037112 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:43.016916 18189 raft_consensus.cc:595] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Received request to transfer leadership to TS 5c9b5062bdfa46af9396f1a85025a331
W20260807 03:39:43.017654 17899 catalog_manager.cc:5018] ChangeConfig:REMOVE_PEER RPC for tablet a21f43f2cbf24aeca08087d9e2fb9b2a with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER failed with leader 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 6 but the committed config has opid_index of 7
I20260807 03:39:43.004068 22427 consensus_queue.cc:260] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:43.012333 18248 leader_election.cc:302] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:43.024864 18312 raft_consensus.cc:1261] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Refusing update from remote peer 5c9b5062bdfa46af9396f1a85025a331: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 03:39:43.007736 17918 rpcz_store.cc:276] 0807 03:39:39.272131 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:39.272269 (+ 138us) service_pool.cc:227] Handling call
0807 03:39:43.007222 (+3734953us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 03:39:41.593747 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 03:39:41.609940 (+ 16193us) write_op.cc:432] Acquiring schema lock in shared mode
0807 03:39:41.609984 (+ 44us) write_op.cc:435] Acquired schema lock
0807 03:39:41.610002 (+ 18us) tablet.cc:660] Decoding operations
0807 03:39:41.610354 (+ 352us) write_op.cc:620] Acquiring the partition lock for write op
0807 03:39:41.610447 (+ 93us) write_op.cc:641] Partition lock acquired for write op
0807 03:39:41.610478 (+ 31us) tablet.cc:683] Acquiring locks for 1 operations
0807 03:39:41.610678 (+ 200us) tablet.cc:699] Row locks acquired
0807 03:39:41.610701 (+ 23us) write_op.cc:260] PREPARE: finished
0807 03:39:41.610849 (+ 148us) write_op.cc:270] Start()
0807 03:39:41.610970 (+ 121us) write_op.cc:276] Timestamp: P: 1786073981610802 usec, L: 0
0807 03:39:41.610993 (+ 23us) op_driver.cc:331] REPLICATION: starting
0807 03:39:41.611659 (+ 666us) log.cc:842] Serialized 544 byte log entry
0807 03:39:42.012777 (+401118us) op_driver.cc:444] REPLICATION: finished
0807 03:39:42.562057 (+549280us) write_op.cc:301] APPLY: starting
0807 03:39:42.570538 (+ 8481us) tablet.cc:1365] starting BulkCheckPresence
0807 03:39:42.586083 (+ 15545us) tablet.cc:1368] finished BulkCheckPresence
0807 03:39:42.586269 (+ 186us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 03:39:42.586584 (+ 315us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 03:39:42.586856 (+ 272us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0807 03:39:42.586929 (+ 73us) write_op.cc:312] APPLY: finished
0807 03:39:42.587176 (+ 247us) log.cc:842] Serialized 23 byte log entry
0807 03:39:42.587375 (+ 199us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:39:42.587478 (+ 103us) write_op.cc:454] Released schema lock
0807 03:39:42.587527 (+ 49us) write_op.cc:341] FINISH: Updating metrics
0807 03:39:42.587924 (+ 397us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:39:42.587955 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":488130,"apply.run_cpu_time_us":2576,"apply.run_wall_time_us":51136,"num_ops":1,"prepare.queue_time_us":216789,"prepare.run_cpu_time_us":16693,"prepare.run_wall_time_us":316569,"raft.queue_time_us":127876,"raft.run_cpu_time_us":35321,"raft.run_wall_time_us":194095,"replication_time_us":401686,"thread_start_us":669027,"threads_started":4,"wal-append.queue_time_us":271173}]]}
I20260807 03:39:43.024046 18190 raft_consensus.cc:2454] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 4.
I20260807 03:39:43.027233 22446 leader_election.cc:288] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:43.027441 22367 raft_consensus.cc:504] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:43.040764 18185 raft_consensus.cc:1226] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 3. Current term is 4. Ops: []
W20260807 03:39:43.040664 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 68070us
W20260807 03:39:43.040848 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 57893.7us
I20260807 03:39:43.044123 18027 leader_election.cc:302] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:43.044188 18141 leader_election.cc:302] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:43.043409 18248 leader_election.cc:302] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:43.049652 18027 leader_election.cc:302] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:43.050362 18248 leader_election.cc:302] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:43.052472 18140 leader_election.cc:302] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:43.052731 18248 leader_election.cc:302] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:43.054114 18187 raft_consensus.cc:1226] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 4. Current term is 5. Ops: []
I20260807 03:39:43.063122 18027 leader_election.cc:302] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:43.063107 18140 leader_election.cc:302] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:43.064626 22456 raft_consensus.cc:2781] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 03:39:43.064827 21958 raft_consensus.cc:2726] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:43.065200 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260807 03:39:43.041810 22288 raft_consensus.cc:2726] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260807 03:39:43.065958 22285 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:33238 (request call id 205) took 2134 ms. Trace:
I20260807 03:39:43.067063 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:39:43.058528 22449 raft_consensus.cc:2726] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:43.042580 22411 raft_consensus.cc:2726] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 03:39:43.068564 22257 raft_consensus.cc:2726] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260807 03:39:43.057984 22446 consensus_queue.cc:1046] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20260807 03:39:43.045334 22472 raft_consensus.cc:2726] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260807 03:39:43.069881 22446 raft_consensus.cc:3032] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 03:39:43.070431 22446 raft_consensus.cc:729] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:43.070498 22298 raft_consensus.cc:2726] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 03:39:43.069926 22401 raft_consensus.cc:2726] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:43.071408 22446 consensus_queue.cc:260] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:43.074229 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:43.074539 22446 raft_consensus.cc:3037] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:43.066952 22285 rpcz_store.cc:276] 0807 03:39:40.928700 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:40.931318 (+ 2618us) service_pool.cc:227] Handling call
0807 03:39:41.422624 (+491306us) spinlock_profiling.cc:243] Waited 238 ms on lock 0x7b18002537a0. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921fa94d 00007fe9921c5f84 00007fe9921c5743 00007fe9921bdc1a 00007fe9971016d9 00007fe9907eb184 00007fe9907eb112 00007fe9907eb041 00007fe9907eafbc 00007fe9907ea272
0807 03:39:41.447626 (+ 25002us) log.cc:842] Serialized 508 byte log entry
0807 03:39:43.062900 (+1615274us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":820,"raft.run_cpu_time_us":1041,"raft.run_wall_time_us":5806,"spinlock_wait_cycles":547980544,"thread_start_us":353016,"threads_started":1,"wal-append.queue_time_us":352437,"wal-append.run_cpu_time_us":1627,"wal-append.run_wall_time_us":1008491}
I20260807 03:39:43.067914 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:39:43.072273 22456 raft_consensus.cc:686] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:43.077425 22456 consensus_queue.cc:237] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:43.055490 22367 leader_election.cc:288] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:43.056124 22470 raft_consensus.cc:2726] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:43.058363 22427 raft_consensus.cc:3037] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:43.085342 22257 raft_consensus.cc:2726] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:43.090183 18248 leader_election.cc:302] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:43.098968 18248 leader_election.cc:302] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:43.099938 17898 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet b15fdfae36e94837adea3c6b66864b0e with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 03:39:43.100842 18248 leader_election.cc:302] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:43.088300 22257 raft_consensus.cc:482] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:43.108407 18248 leader_election.cc:302] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:43.097986 22493 raft_consensus.cc:2726] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260807 03:39:43.111369 22532 raft_consensus.cc:2726] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 03:39:43.111191 22257 raft_consensus.cc:504] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:43.114197 22523 raft_consensus.cc:2726] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 03:39:43.109760 22470 raft_consensus.cc:2781] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260807 03:39:43.123492 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 60447.8us
I20260807 03:39:43.124307 18027 leader_election.cc:302] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:43.125746 18027 leader_election.cc:302] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:43.104689 22349 raft_consensus.cc:2781] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:39:43.128538 18027 leader_election.cc:302] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:43.089707 22519 raft_consensus.cc:504] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:43.134644 18248 leader_election.cc:302] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:43.099670 22371 consensus_queue.cc:1046] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.028s
I20260807 03:39:43.137691 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:43.104341 22472 consensus_queue.cc:1046] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20260807 03:39:43.139874 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:43.105242 22449 consensus_queue.cc:1046] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20260807 03:39:43.126659 22470 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:43.143855 22443 consensus_queue.cc:1035] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 03:39:43.145373 22257 leader_election.cc:288] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:43.128314 22349 raft_consensus.cc:482] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:43.146698 22349 raft_consensus.cc:3037] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:43.143872 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:43.134845 22401 raft_consensus.cc:2726] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:43.149930 18311 raft_consensus.cc:1226] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 6. Current term is 7. Ops: []
I20260807 03:39:43.136175 22288 raft_consensus.cc:2726] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260807 03:39:43.135871 22520 raft_consensus.cc:2726] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 03:39:43.163002 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:43.155054 18140 leader_election.cc:302] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:43.155947 18081 raft_consensus.cc:1226] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:43.156147 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:43.147630 22470 raft_consensus.cc:3037] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 11 FOLLOWER]: Advancing to term 12
I20260807 03:39:43.158277 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:43.154237 18310 raft_consensus.cc:2454] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 8.
I20260807 03:39:43.162225 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:43.138918 22411 raft_consensus.cc:2726] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 03:39:43.168649 18082 raft_consensus.cc:2379] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 8: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:43.169250 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:43.169718 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:43.171312 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:43.175208 22519 leader_election.cc:288] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:43.176975 22520 raft_consensus.cc:3032] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:39:43.177234 22371 raft_consensus.cc:2726] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 03:39:43.177381 22520 raft_consensus.cc:729] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:43.178256 22520 consensus_queue.cc:260] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:43.179437 22520 raft_consensus.cc:3037] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:43.179545 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:43.185189 22470 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:43.187606 18305 raft_consensus.cc:2454] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 4.
I20260807 03:39:43.188311 18300 raft_consensus.cc:1226] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 12 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 11. Current term is 12. Ops: []
I20260807 03:39:43.189845 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:43.196259 18141 leader_election.cc:302] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:43.197726 18141 leader_election.cc:302] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:43.198426 22371 raft_consensus.cc:2726] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260807 03:39:43.198954 22371 raft_consensus.cc:2726] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 03:39:43.201642 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:43.201975 22470 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 12 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:43.202895 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:43.204018 22476 raft_consensus.cc:3032] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Stepping down as leader of term 6
I20260807 03:39:43.204195 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:43.204463 22476 raft_consensus.cc:729] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:43.205399 22476 consensus_queue.cc:260] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:43.206780 22476 raft_consensus.cc:3037] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:43.207073 22349 raft_consensus.cc:504] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:43.223937 18027 leader_election.cc:302] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:43.242108 18140 leader_election.cc:302] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:43.242563 21958 raft_consensus.cc:3037] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:43.245884 18027 leader_election.cc:302] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:43.254040 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:43.262081 21958 raft_consensus.cc:2726] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:43.270562 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:43.277217 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:43.278117 22349 leader_election.cc:288] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163)
I20260807 03:39:43.269169 22371 raft_consensus.cc:2726] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:43.260917 22401 raft_consensus.cc:2781] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader election won for term 4
I20260807 03:39:43.285810 22401 raft_consensus.cc:686] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:43.286911 22401 consensus_queue.cc:237] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:43.270943 18248 leader_election.cc:302] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:43.288684 22550 raft_consensus.cc:2726] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:43.290453 18248 leader_election.cc:302] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:43.291441 18189 raft_consensus.cc:1226] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:43.307727 22427 consensus_queue.cc:1046] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 03:39:43.308808 22427 raft_consensus.cc:3032] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 03:39:43.309144 22427 raft_consensus.cc:729] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:43.309957 22427 consensus_queue.cc:260] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:43.311097 22427 raft_consensus.cc:3037] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:43.311385 17920 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39510 (request call id 197) took 1333 ms. Trace:
I20260807 03:39:43.300848 22550 raft_consensus.cc:2726] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:43.313107 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:43.313462 22493 raft_consensus.cc:3032] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 03:39:43.313884 22493 raft_consensus.cc:729] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:43.314649 22493 consensus_queue.cc:260] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:43.316713 18309 raft_consensus.cc:1226] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 6. Current term is 7. Ops: []
I20260807 03:39:43.321170 17915 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35098 (request call id 77) took 3171 ms. Trace:
I20260807 03:39:43.323930 17912 catalog_manager.cc:5495] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: config changed from index 4 to 5, VOTER 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) evicted. New cstate: current_term: 3 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 03:39:43.311894 17920 rpcz_store.cc:276] 0807 03:39:41.977812 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:41.979000 (+ 1188us) service_pool.cc:227] Handling call
0807 03:39:43.311254 (+1332254us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":164376,"reported_tablets":3}
I20260807 03:39:43.315910 22493 raft_consensus.cc:3037] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:43.294443 18248 leader_election.cc:302] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:43.335197 22520 raft_consensus.cc:2726] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260807 03:39:43.326009 17915 rpcz_store.cc:276] 0807 03:39:40.172372 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:40.172518 (+ 146us) service_pool.cc:227] Handling call
0807 03:39:43.321096 (+3148578us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 03:39:43.073604 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 03:39:43.074154 (+ 550us) write_op.cc:432] Acquiring schema lock in shared mode
0807 03:39:43.074193 (+ 39us) write_op.cc:435] Acquired schema lock
0807 03:39:43.074209 (+ 16us) tablet.cc:660] Decoding operations
0807 03:39:43.074564 (+ 355us) write_op.cc:620] Acquiring the partition lock for write op
0807 03:39:43.074646 (+ 82us) write_op.cc:641] Partition lock acquired for write op
0807 03:39:43.074671 (+ 25us) tablet.cc:683] Acquiring locks for 1 operations
0807 03:39:43.074850 (+ 179us) tablet.cc:699] Row locks acquired
0807 03:39:43.074867 (+ 17us) write_op.cc:260] PREPARE: finished
0807 03:39:43.074995 (+ 128us) write_op.cc:270] Start()
0807 03:39:43.075093 (+ 98us) write_op.cc:276] Timestamp: P: 1786073983074952 usec, L: 0
0807 03:39:43.075112 (+ 19us) op_driver.cc:331] REPLICATION: starting
0807 03:39:43.075740 (+ 628us) log.cc:842] Serialized 539 byte log entry
0807 03:39:43.147148 (+ 71408us) op_driver.cc:444] REPLICATION: finished
0807 03:39:43.267006 (+119858us) write_op.cc:301] APPLY: starting
0807 03:39:43.267069 (+ 63us) tablet.cc:1365] starting BulkCheckPresence
0807 03:39:43.267141 (+ 72us) tablet.cc:1368] finished BulkCheckPresence
0807 03:39:43.267238 (+ 97us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 03:39:43.289052 (+ 21814us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 03:39:43.289244 (+ 192us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0807 03:39:43.289286 (+ 42us) write_op.cc:312] APPLY: finished
0807 03:39:43.289494 (+ 208us) log.cc:842] Serialized 23 byte log entry
0807 03:39:43.289660 (+ 166us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:39:43.289744 (+ 84us) write_op.cc:454] Released schema lock
0807 03:39:43.289782 (+ 38us) write_op.cc:341] FINISH: Updating metrics
0807 03:39:43.290138 (+ 356us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:39:43.298027 (+ 7889us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":497044,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":119611,"num_ops":1,"prepare.queue_time_us":87423,"prepare.run_cpu_time_us":2507,"prepare.run_wall_time_us":22769,"replication_time_us":71940,"thread_start_us":150251,"threads_started":2}]]}
I20260807 03:39:43.338037 18248 leader_election.cc:302] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:43.341451 18027 leader_election.cc:302] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:43.353026 21958 raft_consensus.cc:2726] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260807 03:39:43.357117 18141 leader_election.cc:302] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:43.357005 22530 consensus_queue.cc:1046] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:39:43.358249 22367 raft_consensus.cc:3032] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Stepping down as leader of term 6
I20260807 03:39:43.358633 22367 raft_consensus.cc:729] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:43.359440 22367 consensus_queue.cc:260] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:43.360924 22367 raft_consensus.cc:3037] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:43.362145 18141 leader_election.cc:302] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:43.364075 18064 tablet_service.cc:1514] Processing DeleteTablet for tablet a21f43f2cbf24aeca08087d9e2fb9b2a with delete_type TABLET_DATA_TOMBSTONED (TS d9716e2bf06442f5b3b0e81087c53c4d not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33238
I20260807 03:39:43.364290 22446 raft_consensus.cc:2726] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 03:39:43.363328 18141 leader_election.cc:302] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:43.368734 18248 leader_election.cc:302] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:43.364967 22446 raft_consensus.cc:2726] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 03:39:43.379280 22476 raft_consensus.cc:2726] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 03:39:43.380821 18311 raft_consensus.cc:1226] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:43.343703 22520 raft_consensus.cc:2726] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:43.389051 22257 consensus_queue.cc:1046] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 03:39:43.390094 22257 raft_consensus.cc:3032] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 03:39:43.392685 22257 raft_consensus.cc:729] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:43.393509 22257 consensus_queue.cc:260] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:43.394280 22532 raft_consensus.cc:2726] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260807 03:39:43.394769 22257 raft_consensus.cc:3037] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:43.393587 18248 leader_election.cc:302] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:43.395713 22532 raft_consensus.cc:2726] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260807 03:39:43.398401 18300 raft_consensus.cc:2932] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:43.402206 21958 raft_consensus.cc:979] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d: : Instructing follower 66a36e1ac8e945e893668e282d394c49 to start an election
I20260807 03:39:43.402645 21958 raft_consensus.cc:1067] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 LEADER]: Signalling peer 66a36e1ac8e945e893668e282d394c49 to start an election
I20260807 03:39:43.412199 18168 tablet_service.cc:1514] Processing DeleteTablet for tablet a3dc635ee56b4559a87f5955bb83235a with delete_type TABLET_DATA_TOMBSTONED (TS 3c4bceb836db4536a0f255fdf86abc02 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44650
I20260807 03:39:43.406807 17918 catalog_manager.cc:5495] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: config changed from index 6 to 7, VOTER 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) evicted. New cstate: current_term: 5 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:43.415601 17918 catalog_manager.cc:5495] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 3 to 4, leader changed from d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 4 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:43.416850 18027 leader_election.cc:302] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:43.417631 21958 raft_consensus.cc:2726] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 03:39:43.418120 18027 leader_election.cc:302] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:43.419585 18027 leader_election.cc:302] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:43.424834 21958 raft_consensus.cc:2781] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:43.425269 21958 raft_consensus.cc:482] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:43.425629 21958 raft_consensus.cc:3037] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:43.379938 22476 consensus_queue.cc:1046] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.037s
I20260807 03:39:43.432933 18081 consensus_queue.cc:237] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:43.442355 22546 raft_consensus.cc:482] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:43.442967 22546 raft_consensus.cc:504] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:43.445798 22546 leader_election.cc:288] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:43.447469 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:43.451064 18248 leader_election.cc:302] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:43.454263 22559 tablet_replica.cc:330] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d: stopping tablet replica
I20260807 03:39:43.455415 22559 raft_consensus.cc:2229] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Raft consensus shutting down.
I20260807 03:39:43.430598 22411 raft_consensus.cc:2781] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader election won for term 8
I20260807 03:39:43.455694 21958 raft_consensus.cc:504] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:43.460223 21958 leader_election.cc:288] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:43.460080 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:43.459563 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:43.436540 22528 raft_consensus.cc:482] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:43.465281 18140 leader_election.cc:302] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:43.465924 21958 consensus_queue.cc:1046] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 03:39:43.473517 18311 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4"
dest_uuid: "66a36e1ac8e945e893668e282d394c49"
from {username='slave'} at 127.0.0.1:57276
I20260807 03:39:43.474203 18311 raft_consensus.cc:482] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:39:43.474674 18311 raft_consensus.cc:3037] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:43.460896 22470 raft_consensus.cc:2726] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 03:39:43.476245 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:43.476902 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:43.470064 22409 raft_consensus.cc:2726] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:43.489533 18189 raft_consensus.cc:2379] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 4: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:43.464205 22411 raft_consensus.cc:686] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 8 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:43.476405 19290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:43.539734 18300 raft_consensus.cc:2454] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 4.
I20260807 03:39:43.539279 18311 raft_consensus.cc:504] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:43.548760 22446 raft_consensus.cc:3032] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 11 LEADER]: Stepping down as leader of term 11
I20260807 03:39:43.549219 22446 raft_consensus.cc:729] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:43.550215 22446 consensus_queue.cc:260] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:43.553051 18311 leader_election.cc:288] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:43.554155 22539 raft_consensus.cc:3032] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 LEADER]: Stepping down as leader of term 6
I20260807 03:39:43.557070 22539 raft_consensus.cc:729] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:43.559020 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:43.559676 18070 raft_consensus.cc:3032] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 LEADER]: Stepping down as leader of term 4
I20260807 03:39:43.558164 22539 consensus_queue.cc:260] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:43.560971 22539 raft_consensus.cc:3037] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:43.554445 22562 tablet_replica.cc:330] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02: stopping tablet replica
I20260807 03:39:43.552788 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
W20260807 03:39:43.554234 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 51536.4us
I20260807 03:39:43.567219 18189 raft_consensus.cc:3037] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:43.560006 18070 raft_consensus.cc:729] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:43.564023 22559 raft_consensus.cc:2258] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Raft consensus is shut down!
I20260807 03:39:43.566737 22367 raft_consensus.cc:979] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02: : Instructing follower 5c9b5062bdfa46af9396f1a85025a331 to start an election
I20260807 03:39:43.466601 22528 raft_consensus.cc:504] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:43.572749 18070 consensus_queue.cc:260] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:43.583073 18189 raft_consensus.cc:2454] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:43.594127 18310 raft_consensus.cc:1261] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260807 03:39:43.591584 18249 leader_election.cc:302] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:43.597733 18070 raft_consensus.cc:3037] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:43.597849 22427 raft_consensus.cc:2781] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 03:39:43.599824 22427 raft_consensus.cc:686] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:43.600680 22427 consensus_queue.cc:237] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:43.598731 22498 log.cc:924] Time spent T 00000000000000000000000000000000 P 8da2065a77024f35bdfc0dc3adb83f69: Append to log took a long time: real 0.103s user 0.001s sys 0.000s
I20260807 03:39:43.607609 18027 leader_election.cc:302] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:43.620919 22401 raft_consensus.cc:2781] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader election won for term 4
I20260807 03:39:43.621407 22401 raft_consensus.cc:686] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:43.622201 22481 consensus_queue.cc:1035] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260807 03:39:43.623394 22411 consensus_queue.cc:237] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:43.622203 22401 consensus_queue.cc:237] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:43.595588 22367 raft_consensus.cc:1067] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Signalling peer 5c9b5062bdfa46af9396f1a85025a331 to start an election
I20260807 03:39:43.642889 19290 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a"
dest_uuid: "5c9b5062bdfa46af9396f1a85025a331"
from {username='slave'} at 127.0.0.1:59920
I20260807 03:39:43.643360 22559 ts_tablet_manager.cc:1905] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 03:39:43.643512 19290 raft_consensus.cc:482] T a21f43f2cbf24aeca08087d9e2fb9b2a P 5c9b5062bdfa46af9396f1a85025a331 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:39:43.644006 19290 raft_consensus.cc:3037] T a21f43f2cbf24aeca08087d9e2fb9b2a P 5c9b5062bdfa46af9396f1a85025a331 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:43.649485 22528 leader_election.cc:288] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:43.651576 19297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:43.652318 19297 raft_consensus.cc:2379] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 4: Already voted for candidate 5c9b5062bdfa46af9396f1a85025a331 in this term.
I20260807 03:39:43.654142 18284 tablet_service.cc:1514] Processing DeleteTablet for tablet b15fdfae36e94837adea3c6b66864b0e with delete_type TABLET_DATA_TOMBSTONED (TS 66a36e1ac8e945e893668e282d394c49 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57260
I20260807 03:39:43.654366 19295 raft_consensus.cc:1261] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 5c9b5062bdfa46af9396f1a85025a331 [term 9 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260807 03:39:43.657382 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:43.659904 18070 raft_consensus.cc:2454] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:43.658037 18311 raft_consensus.cc:2454] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:39:43.663771 18141 leader_election.cc:302] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:39:43.678979 18305 raft_consensus.cc:1261] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 03:39:43.678133 17912 catalog_manager.cc:5495] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 1 to 2, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132). New cstate: current_term: 2 leader_uuid: "5c9b5062bdfa46af9396f1a85025a331" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:43.690613 17912 catalog_manager.cc:5495] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 3 to 4, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 4 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:43.672052 22367 raft_consensus.cc:2781] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:43.710107 22562 raft_consensus.cc:2229] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260807 03:39:43.711071 22562 raft_consensus.cc:2258] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 03:39:43.707739 22367 raft_consensus.cc:482] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:43.712671 22367 raft_consensus.cc:3037] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:43.714056 22562 ts_tablet_manager.cc:1905] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 03:39:43.688467 22531 consensus_queue.cc:1035] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.045s
I20260807 03:39:43.671505 22540 consensus_queue.cc:1035] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.047s
I20260807 03:39:43.726070 22132 raft_consensus.cc:482] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:43.757792 22132 raft_consensus.cc:504] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:43.761868 19290 raft_consensus.cc:504] T a21f43f2cbf24aeca08087d9e2fb9b2a P 5c9b5062bdfa46af9396f1a85025a331 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:43.766940 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:43.769237 18184 raft_consensus.cc:2454] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 7.
I20260807 03:39:43.770484 18027 leader_election.cc:302] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:43.772207 18070 raft_consensus.cc:1261] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 03:39:43.773856 22573 tablet_replica.cc:330] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49: stopping tablet replica
I20260807 03:39:43.774112 22285 raft_consensus.cc:2781] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:39:43.774497 22285 raft_consensus.cc:482] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:43.774802 22285 raft_consensus.cc:3037] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:43.907872 19290 leader_election.cc:288] T a21f43f2cbf24aeca08087d9e2fb9b2a P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 7 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:43.909284 22367 raft_consensus.cc:504] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:43.911173 22573 raft_consensus.cc:2229] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260807 03:39:43.859663 22132 leader_election.cc:288] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:43.920776 22531 consensus_queue.cc:1035] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260807 03:39:43.922598 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 136255us
I20260807 03:39:43.928462 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:43.931689 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: true dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:43.932278 18185 raft_consensus.cc:3032] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Stepping down as leader of term 6
I20260807 03:39:43.932526 18185 raft_consensus.cc:729] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:43.933212 18185 consensus_queue.cc:260] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:43.934700 18185 raft_consensus.cc:3037] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:43.935149 18300 raft_consensus.cc:2454] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:39:43.941573 22439 raft_consensus.cc:482] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:43.942096 22439 raft_consensus.cc:504] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:43.944393 22439 leader_election.cc:288] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:43.947130 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:43.947850 18311 raft_consensus.cc:2379] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 7: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:43.948149 18305 raft_consensus.cc:1261] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 03:39:43.953390 22446 raft_consensus.cc:3037] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 11 FOLLOWER]: Advancing to term 12
I20260807 03:39:43.956360 22367 leader_election.cc:288] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Requested vote from peers 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:43.964581 18185 raft_consensus.cc:2454] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 7.
I20260807 03:39:43.965987 19240 leader_election.cc:302] T a21f43f2cbf24aeca08087d9e2fb9b2a P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331; no voters:
I20260807 03:39:43.966845 22507 raft_consensus.cc:2781] T a21f43f2cbf24aeca08087d9e2fb9b2a P 5c9b5062bdfa46af9396f1a85025a331 [term 7 FOLLOWER]: Leader election won for term 7
W20260807 03:39:43.967805 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 183627us
I20260807 03:39:43.970440 22573 raft_consensus.cc:2258] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260807 03:39:43.974092 18184 raft_consensus.cc:1261] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 03:39:43.974651 22580 raft_consensus.cc:482] T d24d0bbea8bd49d482ad95538ca6e631 P 5c9b5062bdfa46af9396f1a85025a331 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:43.975243 22580 raft_consensus.cc:504] T d24d0bbea8bd49d482ad95538ca6e631 P 5c9b5062bdfa46af9396f1a85025a331 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:43.977463 22580 leader_election.cc:288] T d24d0bbea8bd49d482ad95538ca6e631 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:43.979432 22562 ts_tablet_manager.cc:1918] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.4
I20260807 03:39:43.979934 22562 log.cc:1196] T a3dc635ee56b4559a87f5955bb83235a P 3c4bceb836db4536a0f255fdf86abc02: Deleting WAL directory at /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786073845599630-16154-0/minicluster-data/ts-1-root/wals/a3dc635ee56b4559a87f5955bb83235a
I20260807 03:39:43.985925 22573 ts_tablet_manager.cc:1905] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 03:39:43.992421 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:43.993137 18184 raft_consensus.cc:2454] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 4.
I20260807 03:39:43.993438 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:43.994058 18082 raft_consensus.cc:2454] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 5.
I20260807 03:39:43.995257 19239 leader_election.cc:302] T d24d0bbea8bd49d482ad95538ca6e631 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:43.996028 22580 raft_consensus.cc:2781] T d24d0bbea8bd49d482ad95538ca6e631 P 5c9b5062bdfa46af9396f1a85025a331 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:43.996474 22580 raft_consensus.cc:482] T d24d0bbea8bd49d482ad95538ca6e631 P 5c9b5062bdfa46af9396f1a85025a331 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:43.996822 22580 raft_consensus.cc:3037] T d24d0bbea8bd49d482ad95538ca6e631 P 5c9b5062bdfa46af9396f1a85025a331 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:44.006839 22285 raft_consensus.cc:504] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:44.013526 19292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331"
I20260807 03:39:44.014516 19292 raft_consensus.cc:2379] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 4: Already voted for candidate 5c9b5062bdfa46af9396f1a85025a331 in this term.
I20260807 03:39:44.019349 22507 raft_consensus.cc:686] T a21f43f2cbf24aeca08087d9e2fb9b2a P 5c9b5062bdfa46af9396f1a85025a331 [term 7 LEADER]: Becoming Leader. State: Replica: 5c9b5062bdfa46af9396f1a85025a331, State: Running, Role: LEADER
I20260807 03:39:44.011601 22580 raft_consensus.cc:504] T d24d0bbea8bd49d482ad95538ca6e631 P 5c9b5062bdfa46af9396f1a85025a331 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:44.020413 22507 consensus_queue.cc:237] T a21f43f2cbf24aeca08087d9e2fb9b2a P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:44.024338 22580 leader_election.cc:288] T d24d0bbea8bd49d482ad95538ca6e631 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:44.034320 18141 leader_election.cc:302] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:39:44.035668 22476 raft_consensus.cc:2781] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 03:39:44.036279 22476 raft_consensus.cc:686] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:44.037470 17899 catalog_manager.cc:4841] TS 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057): tablet a3dc635ee56b4559a87f5955bb83235a (table test-workload [id=677ca051b50345e183d60556fc42291d]) successfully deleted
I20260807 03:39:44.037652 19240 leader_election.cc:302] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331; no voters:
I20260807 03:39:44.038329 22580 raft_consensus.cc:2781] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:44.039158 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: true dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:44.039747 18300 raft_consensus.cc:3037] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:44.037135 22476 consensus_queue.cc:237] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:44.038691 22580 raft_consensus.cc:482] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:44.044354 22580 raft_consensus.cc:3037] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:44.051379 18300 raft_consensus.cc:2454] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 7.
I20260807 03:39:44.058130 22580 raft_consensus.cc:504] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:44.038705 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:44.060774 18184 raft_consensus.cc:2379] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5c9b5062bdfa46af9396f1a85025a331 in current term 5: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:44.066612 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:44.067703 18300 raft_consensus.cc:2454] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 4.
I20260807 03:39:44.066063 17915 catalog_manager.cc:5495] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 3 to 4, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 4 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:44.067574 17918 catalog_manager.cc:5495] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 7 to 8, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 8 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:44.070691 17918 catalog_manager.cc:5495] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 3 to 4, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 4 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:44.075143 22559 ts_tablet_manager.cc:1918] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 6.6
I20260807 03:39:44.075567 22559 log.cc:1196] T a21f43f2cbf24aeca08087d9e2fb9b2a P d9716e2bf06442f5b3b0e81087c53c4d: Deleting WAL directory at /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786073845599630-16154-0/minicluster-data/ts-0-root/wals/a21f43f2cbf24aeca08087d9e2fb9b2a
I20260807 03:39:44.078143 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:44.078958 18189 raft_consensus.cc:2379] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5c9b5062bdfa46af9396f1a85025a331 in current term 5: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:44.080166 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:44.092339 22411 consensus_queue.cc:1035] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260807 03:39:44.094270 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 90390.6us
I20260807 03:39:44.093904 22411 raft_consensus.cc:2932] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 LEADER]: Committing config change with OpId 9.10: config changed from index 9 to 10, VOTER 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
W20260807 03:39:44.094676 18025 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.132:41381, user_credentials={real_user=slave}} blocked reactor thread for 89395.3us
I20260807 03:39:44.101583 22285 leader_election.cc:288] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:44.107894 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:44.109791 18311 raft_consensus.cc:2932] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Committing config change with OpId 9.10: config changed from index 9 to 10, VOTER 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:44.114184 22462 consensus_queue.cc:1035] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:39:44.113890 17920 catalog_manager.cc:5495] T a21f43f2cbf24aeca08087d9e2fb9b2a P 5c9b5062bdfa46af9396f1a85025a331 reported cstate change: term changed from 6 to 7, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132). New cstate: current_term: 7 leader_uuid: "5c9b5062bdfa46af9396f1a85025a331" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 03:39:44.121805 19292 raft_consensus.cc:2932] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 5c9b5062bdfa46af9396f1a85025a331 [term 9 FOLLOWER]: Committing config change with OpId 9.10: config changed from index 9 to 10, VOTER 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:44.124878 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:44.126874 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:44.127503 18310 raft_consensus.cc:2379] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 7: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:44.135295 18298 raft_consensus.cc:1261] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260807 03:39:44.142226 18189 raft_consensus.cc:2454] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 7.
I20260807 03:39:44.138610 17912 catalog_manager.cc:5495] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 4 to 5, leader changed from d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 5 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:39:44.154449 18071 raft_consensus.cc:2454] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 5.
I20260807 03:39:44.157502 19239 leader_election.cc:302] T d24d0bbea8bd49d482ad95538ca6e631 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:44.179637 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:44.182222 18300 raft_consensus.cc:3037] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:44.162834 22507 raft_consensus.cc:2781] T d24d0bbea8bd49d482ad95538ca6e631 P 5c9b5062bdfa46af9396f1a85025a331 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 03:39:44.160475 22580 leader_election.cc:288] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 5 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:44.204064 18300 raft_consensus.cc:2454] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 5.
I20260807 03:39:44.205682 19240 leader_election.cc:302] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02
W20260807 03:39:44.218101 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 74421us
W20260807 03:39:44.221532 18025 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.132:41381, user_credentials={real_user=slave}} blocked reactor thread for 63484.2us
I20260807 03:39:44.221742 18027 leader_election.cc:302] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:44.231158 18189 raft_consensus.cc:1261] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260807 03:39:44.230470 17898 catalog_manager.cc:4841] TS d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163): tablet a21f43f2cbf24aeca08087d9e2fb9b2a (table test-workload [id=677ca051b50345e183d60556fc42291d]) successfully deleted
I20260807 03:39:44.236644 22588 consensus_queue.cc:1035] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 03:39:44.266893 22550 raft_consensus.cc:482] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:44.267464 22550 raft_consensus.cc:504] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:44.269476 22550 leader_election.cc:288] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:44.272773 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:44.273429 18184 raft_consensus.cc:2454] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:44.274626 18249 leader_election.cc:302] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:44.275454 22427 raft_consensus.cc:2781] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:39:44.275841 22427 raft_consensus.cc:482] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:44.276165 22427 raft_consensus.cc:3037] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:44.226119 22580 raft_consensus.cc:2781] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 03:39:44.276736 22580 raft_consensus.cc:686] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 5 LEADER]: Becoming Leader. State: Replica: 5c9b5062bdfa46af9396f1a85025a331, State: Running, Role: LEADER
I20260807 03:39:44.277633 22580 consensus_queue.cc:237] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:44.281224 18189 raft_consensus.cc:1261] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260807 03:39:44.228402 22401 raft_consensus.cc:2781] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election won for term 7
I20260807 03:39:44.282793 22549 consensus_queue.cc:1035] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 03:39:44.283275 22401 raft_consensus.cc:686] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:44.284231 22401 consensus_queue.cc:237] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:44.305259 17898 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet d8d26ed9a1e340ce8c2d127ac7fbf7f7 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 03:39:44.321004 22427 raft_consensus.cc:504] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:44.325461 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:44.326201 18184 raft_consensus.cc:3037] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:44.334702 18184 raft_consensus.cc:2454] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 6.
I20260807 03:39:44.335974 18249 leader_election.cc:302] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:44.336795 22550 raft_consensus.cc:2781] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 03:39:44.337317 22550 raft_consensus.cc:686] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [term 6 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:44.338130 22550 consensus_queue.cc:237] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:44.349848 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:39:44.365027 22528 raft_consensus.cc:482] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 03:39:44.367149 19240 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 161588us
I20260807 03:39:44.361846 22427 leader_election.cc:288] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
W20260807 03:39:44.377736 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 142339us
I20260807 03:39:44.346110 22507 raft_consensus.cc:686] T d24d0bbea8bd49d482ad95538ca6e631 P 5c9b5062bdfa46af9396f1a85025a331 [term 5 LEADER]: Becoming Leader. State: Replica: 5c9b5062bdfa46af9396f1a85025a331, State: Running, Role: LEADER
I20260807 03:39:44.387480 18184 raft_consensus.cc:1261] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 03:39:44.394577 18081 raft_consensus.cc:1261] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260807 03:39:44.395449 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:44.396016 18070 raft_consensus.cc:3037] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:44.402601 18070 raft_consensus.cc:2454] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 6.
I20260807 03:39:44.365736 22528 raft_consensus.cc:504] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:44.419234 18300 raft_consensus.cc:1261] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 03:39:44.420156 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:44.420732 18072 raft_consensus.cc:2362] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 6. Re-sending same reply.
I20260807 03:39:44.433745 22507 consensus_queue.cc:237] T d24d0bbea8bd49d482ad95538ca6e631 P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:44.446041 22573 ts_tablet_manager.cc:1918] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 5.6
I20260807 03:39:44.427619 22532 consensus_queue.cc:1035] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260807 03:39:44.452855 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:44.415836 22589 raft_consensus.cc:482] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:44.457728 22589 raft_consensus.cc:504] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:44.459824 22589 leader_election.cc:288] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:44.460624 22588 consensus_queue.cc:1035] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 03:39:44.462424 18072 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf"
dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
mode: GRACEFUL
new_leader_uuid: "3c4bceb836db4536a0f255fdf86abc02"
from {username='slave'} at 127.0.0.1:55742
I20260807 03:39:44.473138 18072 raft_consensus.cc:595] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Received request to transfer leadership to TS 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:44.463848 22462 consensus_queue.cc:1035] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 03:39:44.465276 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:44.465780 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:44.451336 22528 leader_election.cc:288] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:44.474540 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:44.478881 18190 raft_consensus.cc:2379] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 5: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:44.478840 18075 raft_consensus.cc:2454] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 6.
I20260807 03:39:44.476553 18309 raft_consensus.cc:2454] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 5.
I20260807 03:39:44.479996 17915 catalog_manager.cc:5495] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 6 to 7, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 7 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:44.481556 17915 catalog_manager.cc:5495] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: config changed from index 9 to 10, VOTER 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) evicted. New cstate: current_term: 9 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 03:39:44.483978 18140 leader_election.cc:302] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:44.485780 22257 raft_consensus.cc:2781] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:39:44.486231 22257 raft_consensus.cc:482] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:44.489853 22257 raft_consensus.cc:3037] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:44.472950 22573 log.cc:1196] T b15fdfae36e94837adea3c6b66864b0e P 66a36e1ac8e945e893668e282d394c49: Deleting WAL directory at /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786073845599630-16154-0/minicluster-data/ts-2-root/wals/b15fdfae36e94837adea3c6b66864b0e
I20260807 03:39:44.498464 18027 leader_election.cc:302] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:44.509595 22411 raft_consensus.cc:2781] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:44.513125 22411 raft_consensus.cc:482] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:44.512995 18072 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6129789267024fac8a337070c10a472a"
dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
mode: GRACEFUL
new_leader_uuid: "3c4bceb836db4536a0f255fdf86abc02"
from {username='slave'} at 127.0.0.1:55742
I20260807 03:39:44.513738 22411 raft_consensus.cc:3037] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:44.513938 18072 raft_consensus.cc:595] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 9 LEADER]: Received request to transfer leadership to TS 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:44.483170 22602 consensus_queue.cc:1035] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20260807 03:39:44.527892 17899 catalog_manager.cc:4841] TS 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021): tablet b15fdfae36e94837adea3c6b66864b0e (table test-workload [id=677ca051b50345e183d60556fc42291d]) successfully deleted
I20260807 03:39:44.540040 18190 raft_consensus.cc:1261] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Refusing update from remote peer 5c9b5062bdfa46af9396f1a85025a331: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260807 03:39:44.544844 18300 raft_consensus.cc:1261] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Refusing update from remote peer 5c9b5062bdfa46af9396f1a85025a331: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260807 03:39:44.554059 19292 raft_consensus.cc:1226] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 5 LEADER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 4. Current term is 5. Ops: []
I20260807 03:39:44.556536 22580 consensus_queue.cc:1035] T a21f43f2cbf24aeca08087d9e2fb9b2a P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260807 03:39:44.556033 22528 consensus_queue.cc:1046] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20260807 03:39:44.557263 22594 consensus_queue.cc:1035] T a21f43f2cbf24aeca08087d9e2fb9b2a P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260807 03:39:44.560379 22528 consensus_queue.cc:1225] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer 5c9b5062bdfa46af9396f1a85025a331 log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260807 03:39:44.567788 22427 raft_consensus.cc:482] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:44.568694 18190 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8c98c1d422654739af06653a512f7499"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
mode: GRACEFUL
new_leader_uuid: "66a36e1ac8e945e893668e282d394c49"
from {username='slave'} at 127.0.0.1:34758
I20260807 03:39:44.568544 22427 raft_consensus.cc:504] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:44.569294 18190 raft_consensus.cc:595] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Received request to transfer leadership to TS 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:44.569772 22367 raft_consensus.cc:3032] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 03:39:44.570195 22367 raft_consensus.cc:729] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:44.571187 22427 leader_election.cc:288] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:44.574824 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:44.575915 18075 raft_consensus.cc:2379] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 6: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:44.577908 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:44.580480 22367 consensus_queue.cc:260] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:44.581961 18184 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bb66518c65e4495e881ce860253d0801"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
mode: GRACEFUL
new_leader_uuid: "66a36e1ac8e945e893668e282d394c49"
from {username='slave'} at 127.0.0.1:34758
I20260807 03:39:44.582213 22367 raft_consensus.cc:3037] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:44.582521 18184 raft_consensus.cc:595] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Received request to transfer leadership to TS 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:44.585700 22411 raft_consensus.cc:504] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:44.588817 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 79327us
I20260807 03:39:44.592828 18185 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
mode: GRACEFUL
new_leader_uuid: "66a36e1ac8e945e893668e282d394c49"
from {username='slave'} at 127.0.0.1:34758
I20260807 03:39:44.593359 18185 raft_consensus.cc:595] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Received request to transfer leadership to TS 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:44.607136 18168 tablet_service.cc:1514] Processing DeleteTablet for tablet d8d26ed9a1e340ce8c2d127ac7fbf7f7 with delete_type TABLET_DATA_TOMBSTONED (TS 3c4bceb836db4536a0f255fdf86abc02 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:44650
I20260807 03:39:44.608335 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260807 03:39:44.610278 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:39:44.587931 22411 leader_election.cc:288] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:44.610924 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:39:44.603485 17912 catalog_manager.cc:5495] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:39:44.616575 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:44.610049 17920 catalog_manager.cc:5495] T d24d0bbea8bd49d482ad95538ca6e631 P 5c9b5062bdfa46af9396f1a85025a331 reported cstate change: term changed from 4 to 5, leader changed from d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) to 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132). New cstate: current_term: 5 leader_uuid: "5c9b5062bdfa46af9396f1a85025a331" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 03:39:44.619423 17920 catalog_manager.cc:5495] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 reported cstate change: term changed from 4 to 5, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132). New cstate: current_term: 5 leader_uuid: "5c9b5062bdfa46af9396f1a85025a331" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 03:39:44.642195 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:44.643302 18184 raft_consensus.cc:2379] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 5: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:44.669911 22257 raft_consensus.cc:504] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:44.671173 18190 raft_consensus.cc:2354] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 6. Current term is 7.
I20260807 03:39:44.677042 22597 raft_consensus.cc:979] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d: : Instructing follower 3c4bceb836db4536a0f255fdf86abc02 to start an election
I20260807 03:39:44.677563 22462 raft_consensus.cc:1067] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Signalling peer 3c4bceb836db4536a0f255fdf86abc02 to start an election
I20260807 03:39:44.681602 22621 tablet_replica.cc:330] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02: stopping tablet replica
I20260807 03:39:44.691211 18298 raft_consensus.cc:2454] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 5.
I20260807 03:39:44.754331 18188 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
from {username='slave'} at 127.0.0.1:44662
W20260807 03:39:44.743665 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 59662.9us
I20260807 03:39:44.792150 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:44.792750 18310 raft_consensus.cc:3037] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 7
I20260807 03:39:44.691774 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:44.796288 18071 raft_consensus.cc:3037] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:44.793645 18311 raft_consensus.cc:1261] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260807 03:39:44.734965 18249 leader_election.cc:398] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057) with higher term. Message: Invalid argument: T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 6. Current term is 7.
I20260807 03:39:44.798656 18249 leader_election.cc:401] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260807 03:39:44.800447 18027 leader_election.cc:302] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:44.776959 22530 raft_consensus.cc:979] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02: : Instructing follower 66a36e1ac8e945e893668e282d394c49 to start an election
I20260807 03:39:44.806694 18310 raft_consensus.cc:2454] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 7.
I20260807 03:39:44.806190 22371 raft_consensus.cc:482] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:44.777261 22411 raft_consensus.cc:482] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:44.807878 22132 raft_consensus.cc:2781] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election won for term 5
I20260807 03:39:44.811250 18188 raft_consensus.cc:482] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
W20260807 03:39:44.812747 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 84171.4us
I20260807 03:39:44.813119 18188 raft_consensus.cc:3037] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:44.671976 22257 leader_election.cc:288] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:44.815177 18071 raft_consensus.cc:2454] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 7.
I20260807 03:39:44.816300 18140 leader_election.cc:302] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:44.817263 18180 raft_consensus.cc:1261] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Refusing update from remote peer 5c9b5062bdfa46af9396f1a85025a331: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260807 03:39:44.817631 18180 pending_rounds.cc:76] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02: Aborting all ops after (but not including) 5
I20260807 03:39:44.819895 22528 raft_consensus.cc:1067] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Signalling peer 66a36e1ac8e945e893668e282d394c49 to start an election
I20260807 03:39:44.820889 22132 raft_consensus.cc:686] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:44.821770 22132 consensus_queue.cc:237] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:44.815676 22549 raft_consensus.cc:2726] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057) with higher term. Message: Invalid argument: T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 6. Current term is 7.
I20260807 03:39:44.817942 18180 pending_rounds.cc:97] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260807 03:39:44.824072 18180 raft_consensus.cc:2866] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 03:39:44.825649 22507 consensus_queue.cc:1035] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 03:39:44.810814 22371 raft_consensus.cc:504] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } attrs { replace: true } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:44.811920 22614 consensus_queue.cc:1035] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
I20260807 03:39:44.829821 22371 leader_election.cc:288] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:44.831290 19292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:44.832383 22621 raft_consensus.cc:2229] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Raft consensus shutting down.
I20260807 03:39:44.832762 18300 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a"
dest_uuid: "66a36e1ac8e945e893668e282d394c49"
from {username='slave'} at 127.0.0.1:57268
I20260807 03:39:44.822785 18298 raft_consensus.cc:1261] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Refusing update from remote peer 5c9b5062bdfa46af9396f1a85025a331: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260807 03:39:44.832904 18188 raft_consensus.cc:504] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:44.835593 22371 raft_consensus.cc:2781] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 03:39:44.835932 22427 raft_consensus.cc:482] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:44.815080 22411 raft_consensus.cc:504] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:44.833400 18300 raft_consensus.cc:482] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:39:44.840374 18300 raft_consensus.cc:3037] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:44.840540 22621 raft_consensus.cc:2258] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Raft consensus is shut down!
I20260807 03:39:44.841521 22411 leader_election.cc:288] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:44.839395 22427 raft_consensus.cc:504] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:44.844403 22371 raft_consensus.cc:686] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:44.844605 22427 leader_election.cc:288] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:44.848881 22371 consensus_queue.cc:237] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:44.861774 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:44.862367 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "66a36e1ac8e945e893668e282d394c49"
W20260807 03:39:44.863047 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 46826.4us
I20260807 03:39:44.863288 18305 raft_consensus.cc:3037] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:44.868774 18141 leader_election.cc:302] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49
W20260807 03:39:44.869582 18141 raft_consensus.cc:2670] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260807 03:39:44.870645 18183 raft_consensus.cc:1261] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 03:39:44.872275 22621 ts_tablet_manager.cc:1905] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 03:39:44.874037 18188 leader_election.cc:288] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:44.875003 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:44.877409 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:44.877883 18075 raft_consensus.cc:3032] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Stepping down as leader of term 5
I20260807 03:39:44.879750 18305 raft_consensus.cc:2454] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 6.
I20260807 03:39:44.888257 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:44.888859 18082 raft_consensus.cc:1261] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Refusing update from remote peer 5c9b5062bdfa46af9396f1a85025a331: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260807 03:39:44.890865 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:44.895180 18191 raft_consensus.cc:2454] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 6.
I20260807 03:39:44.891279 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:44.896479 18310 raft_consensus.cc:2454] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 6.
I20260807 03:39:44.894541 18072 raft_consensus.cc:2454] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:44.878180 18075 raft_consensus.cc:729] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:44.899878 18141 leader_election.cc:302] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:44.901428 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:44.902208 18187 raft_consensus.cc:2379] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 6: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:44.903502 22581 consensus_queue.cc:1035] T d24d0bbea8bd49d482ad95538ca6e631 P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.002s
I20260807 03:39:44.905462 22530 raft_consensus.cc:2781] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 03:39:44.898937 18075 consensus_queue.cc:260] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:44.916898 18075 raft_consensus.cc:3037] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:44.923228 18249 leader_election.cc:302] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:44.924314 22479 consensus_queue.cc:1035] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:39:44.925927 22479 raft_consensus.cc:2781] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:39:44.926286 22479 raft_consensus.cc:482] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:44.905966 22530 raft_consensus.cc:686] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:44.906243 22439 consensus_queue.cc:1035] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.021s
I20260807 03:39:44.927615 22530 consensus_queue.cc:237] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:44.926605 22479 raft_consensus.cc:3037] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:44.919883 18300 raft_consensus.cc:504] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:44.944082 18075 raft_consensus.cc:2454] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 6.
I20260807 03:39:44.947052 18300 leader_election.cc:288] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:44.948504 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:44.949043 18189 raft_consensus.cc:1261] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Refusing update from remote peer 5c9b5062bdfa46af9396f1a85025a331: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260807 03:39:44.950914 22581 consensus_queue.cc:1035] T d24d0bbea8bd49d482ad95538ca6e631 P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260807 03:39:44.952849 22476 raft_consensus.cc:482] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:44.953465 22476 raft_consensus.cc:504] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:44.949074 18187 raft_consensus.cc:3032] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 03:39:44.957129 18187 raft_consensus.cc:729] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:44.957090 19292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:44.958705 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:44.960037 18075 raft_consensus.cc:3037] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:44.958289 18187 consensus_queue.cc:260] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:44.966123 18187 raft_consensus.cc:3037] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:44.974133 18075 raft_consensus.cc:2454] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:44.946372 22441 raft_consensus.cc:482] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:44.955935 22476 leader_election.cc:288] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:44.971766 22462 raft_consensus.cc:482] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:45.002296 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:44.994753 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 46231.9us
I20260807 03:39:45.003233 22367 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:44.984617 22441 raft_consensus.cc:504] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:45.044175 18187 raft_consensus.cc:2454] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:45.049412 18189 raft_consensus.cc:1261] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260807 03:39:45.058713 18027 leader_election.cc:302] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:44.988792 22409 raft_consensus.cc:482] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:45.102344 17918 catalog_manager.cc:5495] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 4 to 5, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129). New cstate: current_term: 5 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:45.047143 22473 raft_consensus.cc:482] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:45.065513 22532 raft_consensus.cc:482] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:45.100907 22411 raft_consensus.cc:2781] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:39:45.150043 22285 raft_consensus.cc:482] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:45.151432 22285 raft_consensus.cc:504] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:45.153371 22285 leader_election.cc:288] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:45.096623 22367 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:45.065724 22462 raft_consensus.cc:504] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:45.101701 22409 raft_consensus.cc:504] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:45.157799 22462 leader_election.cc:288] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:45.160908 22479 raft_consensus.cc:504] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:45.161427 22532 raft_consensus.cc:504] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:45.163293 22532 leader_election.cc:288] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:45.150699 22411 raft_consensus.cc:482] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:45.164150 22411 raft_consensus.cc:3037] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:45.146674 22473 raft_consensus.cc:504] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:45.184522 22479 leader_election.cc:288] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
W20260807 03:39:45.221911 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 159489us
I20260807 03:39:45.218603 18140 leader_election.cc:302] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:45.227541 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:45.230003 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:45.228159 18300 raft_consensus.cc:2454] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 6.
I20260807 03:39:45.232435 18080 raft_consensus.cc:1261] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 03:39:45.230819 18311 raft_consensus.cc:2454] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 7.
I20260807 03:39:45.236346 18249 leader_election.cc:302] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:45.236413 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:45.237390 18189 raft_consensus.cc:2454] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 7.
I20260807 03:39:45.252200 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:45.253198 18189 raft_consensus.cc:3037] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:45.278853 18189 raft_consensus.cc:2454] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 7.
W20260807 03:39:45.280651 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 44418.4us
W20260807 03:39:45.287637 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 69115.3us
W20260807 03:39:45.292184 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 45018.1us
I20260807 03:39:45.302579 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:45.304296 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:45.305294 18072 raft_consensus.cc:2379] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 7: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
W20260807 03:39:45.376492 18025 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.132:41381, user_credentials={real_user=slave}} blocked reactor thread for 149239us
I20260807 03:39:45.372732 22371 raft_consensus.cc:2726] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:45.362818 22644 raft_consensus.cc:2781] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election won for term 5
W20260807 03:39:45.416785 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 190657us
I20260807 03:39:45.415446 22658 raft_consensus.cc:482] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:45.460523 22658 raft_consensus.cc:504] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:45.461553 22621 ts_tablet_manager.cc:1918] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 9.9
I20260807 03:39:45.462018 22621 log.cc:1196] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 3c4bceb836db4536a0f255fdf86abc02: Deleting WAL directory at /tmp/dist-test-taskaL12Z8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786073845599630-16154-0/minicluster-data/ts-1-root/wals/d8d26ed9a1e340ce8c2d127ac7fbf7f7
I20260807 03:39:45.462620 22658 leader_election.cc:288] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:45.512962 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 57715.9us
I20260807 03:39:45.386869 22532 consensus_queue.cc:1035] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.046s
I20260807 03:39:45.433130 22669 raft_consensus.cc:482] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:45.546737 22669 raft_consensus.cc:504] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:45.529974 22132 consensus_queue.cc:1035] T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.020s
I20260807 03:39:45.554466 22669 leader_election.cc:288] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163)
W20260807 03:39:45.556775 22654 log.cc:924] Time spent T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.135s user 0.000s sys 0.000s
W20260807 03:39:45.558655 22643 log.cc:924] Time spent T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.273s user 0.000s sys 0.001s
I20260807 03:39:45.421725 22285 raft_consensus.cc:482] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:45.571509 22285 raft_consensus.cc:504] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:45.433470 22644 raft_consensus.cc:686] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:45.578439 22644 consensus_queue.cc:237] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:45.432883 22411 raft_consensus.cc:504] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:45.582320 22411 leader_election.cc:288] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:45.588748 18071 raft_consensus.cc:3037] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:45.574308 22285 leader_election.cc:288] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:45.605851 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 317334us
I20260807 03:39:45.587906 22473 leader_election.cc:288] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:45.609730 18071 raft_consensus.cc:2454] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 7.
W20260807 03:39:45.719898 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 437828us
W20260807 03:39:45.935848 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 407341us
I20260807 03:39:45.980861 18249 leader_election.cc:302] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:46.003697 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:46.008373 18180 raft_consensus.cc:2454] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 7.
I20260807 03:39:46.044517 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:46.045372 18180 raft_consensus.cc:2454] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 6.
I20260807 03:39:46.047444 22689 raft_consensus.cc:482] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:46.047984 22689 raft_consensus.cc:504] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:46.050287 22689 leader_election.cc:288] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:46.050671 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:46.051456 18082 raft_consensus.cc:2454] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:46.054939 22285 raft_consensus.cc:482] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:46.055526 22285 raft_consensus.cc:504] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:46.058316 22285 leader_election.cc:288] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:46.031683 22411 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:46.075861 18027 leader_election.cc:302] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:46.080569 18027 leader_election.cc:302] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:46.002156 18080 consensus_queue.cc:237] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:39:46.069975 22644 raft_consensus.cc:2781] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260807 03:39:46.120757 18142 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.527s user 0.010s sys 0.024s
W20260807 03:39:46.125721 18142 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.850s user 0.011s sys 0.025s
I20260807 03:39:46.132254 17899 catalog_manager.cc:4841] TS 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057): tablet d8d26ed9a1e340ce8c2d127ac7fbf7f7 (table test-workload [id=677ca051b50345e183d60556fc42291d]) successfully deleted
W20260807 03:39:46.133394 18246 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.597s user 0.017s sys 0.020s
W20260807 03:39:46.133738 18246 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.850s user 0.018s sys 0.020s
I20260807 03:39:46.112596 22401 raft_consensus.cc:482] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:46.131145 22602 raft_consensus.cc:2781] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 03:39:46.152999 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:46.157510 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:46.153239 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:46.120702 22132 raft_consensus.cc:482] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:46.115720 22644 raft_consensus.cc:482] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:46.111693 22686 raft_consensus.cc:2781] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 03:39:46.151929 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:46.166540 18075 raft_consensus.cc:2454] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 5.
I20260807 03:39:46.174302 18075 raft_consensus.cc:1261] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260807 03:39:46.167603 18070 raft_consensus.cc:1261] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260807 03:39:46.112272 22411 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:46.174368 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:46.171168 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:46.174217 18072 raft_consensus.cc:2454] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 7.
I20260807 03:39:46.169201 18249 leader_election.cc:302] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:46.191558 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:46.193734 18305 raft_consensus.cc:2454] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 5.
I20260807 03:39:46.172025 22602 raft_consensus.cc:482] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:46.201440 18069 raft_consensus.cc:2454] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 11.
I20260807 03:39:46.173995 22644 raft_consensus.cc:3037] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:46.172928 22401 raft_consensus.cc:504] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:46.211719 18298 raft_consensus.cc:1261] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260807 03:39:46.213219 22401 leader_election.cc:288] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:46.213796 22693 log.cc:924] Time spent T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20260807 03:39:46.200035 22683 raft_consensus.cc:482] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:46.216620 22683 raft_consensus.cc:504] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:46.196120 22701 raft_consensus.cc:482] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:46.217904 22701 raft_consensus.cc:504] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:46.218510 22683 leader_election.cc:288] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:46.183599 22686 raft_consensus.cc:482] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:46.166168 22685 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:46.228410 22614 raft_consensus.cc:979] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d: : Instructing follower 3c4bceb836db4536a0f255fdf86abc02 to start an election
I20260807 03:39:46.234025 22614 raft_consensus.cc:1067] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 9 LEADER]: Signalling peer 3c4bceb836db4536a0f255fdf86abc02 to start an election
I20260807 03:39:46.184013 22682 raft_consensus.cc:482] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
W20260807 03:39:46.239612 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 96545.7us
I20260807 03:39:46.242152 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:46.242751 18189 raft_consensus.cc:2454] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 6.
I20260807 03:39:46.243189 22581 raft_consensus.cc:482] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 5c9b5062bdfa46af9396f1a85025a331 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:46.243676 22581 raft_consensus.cc:504] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 5c9b5062bdfa46af9396f1a85025a331 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:46.221580 22602 raft_consensus.cc:3037] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Advancing to term 8
W20260807 03:39:46.247176 18025 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.132:41381, user_credentials={real_user=slave}} blocked reactor thread for 116758us
I20260807 03:39:46.182363 22132 raft_consensus.cc:504] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:46.250270 18300 raft_consensus.cc:1261] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260807 03:39:46.253553 22614 raft_consensus.cc:482] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:46.254132 22614 raft_consensus.cc:504] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:46.256479 22614 leader_election.cc:288] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:46.258963 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:46.220274 22701 leader_election.cc:288] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:46.231349 22686 raft_consensus.cc:3037] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Advancing to term 8
W20260807 03:39:46.283627 19240 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 320918us
I20260807 03:39:46.292369 22701 raft_consensus.cc:482] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 03:39:46.451129 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 179191us
W20260807 03:39:46.465227 18025 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.132:41381, user_credentials={real_user=slave}} blocked reactor thread for 189053us
W20260807 03:39:46.490772 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 201940us
I20260807 03:39:46.294008 22705 raft_consensus.cc:2781] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 03:39:46.238744 22685 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:46.245572 22682 raft_consensus.cc:504] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:46.357074 22701 raft_consensus.cc:504] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:46.502429 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:46.549544 22409 leader_election.cc:288] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:46.575304 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 107024us
W20260807 03:39:46.522408 19240 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 206925us
I20260807 03:39:46.551301 22581 leader_election.cc:288] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:46.660236 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:46.660768 18184 raft_consensus.cc:1261] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260807 03:39:46.664052 18185 raft_consensus.cc:2454] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 6.
I20260807 03:39:46.674008 18184 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6129789267024fac8a337070c10a472a"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
from {username='slave'} at 127.0.0.1:44662
I20260807 03:39:46.675413 18184 raft_consensus.cc:482] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:39:46.676003 18184 raft_consensus.cc:3037] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Advancing to term 10
I20260807 03:39:46.676532 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:46.677361 18190 raft_consensus.cc:2354] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 6. Current term is 7.
I20260807 03:39:46.615204 22476 raft_consensus.cc:482] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:46.687433 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:46.688243 18298 raft_consensus.cc:2379] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 6: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:46.687875 22476 raft_consensus.cc:504] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:46.688318 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:46.689668 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:46.690042 22476 leader_election.cc:288] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:46.690270 18293 raft_consensus.cc:2454] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 6.
I20260807 03:39:46.690625 22476 raft_consensus.cc:482] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:46.662647 22411 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:46.647720 22705 raft_consensus.cc:686] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:46.692495 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:46.673631 22651 log.cc:924] Time spent T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 1.259s user 0.008s sys 0.006s
I20260807 03:39:46.704767 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:46.705879 18298 raft_consensus.cc:2379] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 12: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:46.691159 22476 raft_consensus.cc:504] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:46.694657 22685 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:46.706619 22473 raft_consensus.cc:482] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:46.745322 22685 raft_consensus.cc:482] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
W20260807 03:39:46.784641 19240 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 114876us
I20260807 03:39:46.758574 22473 raft_consensus.cc:504] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:46.542104 22715 raft_consensus.cc:482] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:46.627511 22703 raft_consensus.cc:482] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:46.783808 22685 raft_consensus.cc:504] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:46.760637 22705 consensus_queue.cc:237] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:46.783172 22285 raft_consensus.cc:482] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:46.792219 22132 leader_election.cc:288] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:46.801527 22710 log.cc:924] Time spent T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.269s user 0.001s sys 0.001s
I20260807 03:39:46.793291 22614 raft_consensus.cc:482] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:46.713667 22739 raft_consensus.cc:482] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:46.820451 18184 raft_consensus.cc:504] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:46.801574 22740 raft_consensus.cc:482] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:46.833156 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:46.833425 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:46.834892 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:46.835323 19292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: true dest_uuid: "5c9b5062bdfa46af9396f1a85025a331"
I20260807 03:39:46.836090 19292 raft_consensus.cc:3037] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [term 9 FOLLOWER]: Advancing to term 10
I20260807 03:39:46.835186 18184 leader_election.cc:288] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:46.819626 22581 raft_consensus.cc:482] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:46.819727 22754 raft_consensus.cc:482] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5c9b5062bdfa46af9396f1a85025a331)
I20260807 03:39:46.842259 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:46.845530 18293 raft_consensus.cc:2379] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 7: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:46.821743 22701 leader_election.cc:288] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:46.834326 22401 raft_consensus.cc:482] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:46.836086 22682 leader_election.cc:288] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:46.841956 22753 raft_consensus.cc:482] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5c9b5062bdfa46af9396f1a85025a331)
I20260807 03:39:46.824203 22285 raft_consensus.cc:504] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:46.845813 22476 leader_election.cc:288] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:46.878430 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 166946us
I20260807 03:39:46.830708 22614 raft_consensus.cc:504] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:46.814541 22715 raft_consensus.cc:504] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:46.838090 22740 raft_consensus.cc:504] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:46.863476 22581 raft_consensus.cc:504] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:46.819795 22703 raft_consensus.cc:504] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:46.831382 22739 raft_consensus.cc:504] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:46.867345 22754 raft_consensus.cc:504] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:46.922601 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:39:47.279126 19240 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 223128us
I20260807 03:39:47.277611 19292 raft_consensus.cc:2454] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 10.
I20260807 03:39:46.917291 18249 leader_election.cc:302] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:46.953306 22685 leader_election.cc:288] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:46.947484 22462 raft_consensus.cc:482] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:46.952272 22737 raft_consensus.cc:482] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:46.976428 22589 raft_consensus.cc:482] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:46.960335 22285 leader_election.cc:288] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:46.937600 22752 log.cc:924] Time spent T c7a9d935105d41829202c4c161ff09e3 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.084s user 0.000s sys 0.000s
W20260807 03:39:46.936621 18071 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33254 (request call id 2225) took 1624 ms (client timeout 2152 ms). Trace:
I20260807 03:39:46.880035 22682 raft_consensus.cc:482] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
W20260807 03:39:47.360788 18071 rpcz_store.cc:269] 0807 03:39:45.302062 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:45.302288 (+ 226us) service_pool.cc:227] Handling call
0807 03:39:46.924470 (+1622182us) spinlock_profiling.cc:243] Waited 285 ms on lock 0x7b5400033754. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:46.924588 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":656331520}
W20260807 03:39:47.014900 22644 consensus_meta.cc:287] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.779s user 0.005s sys 0.004s
W20260807 03:39:47.026786 22686 consensus_meta.cc:287] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 0.740s user 0.008s sys 0.002s
I20260807 03:39:46.965999 22762 raft_consensus.cc:482] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
W20260807 03:39:46.955380 22602 consensus_meta.cc:287] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 0.685s user 0.008s sys 0.000s
I20260807 03:39:46.949605 22473 leader_election.cc:288] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:46.920764 22367 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:46.962157 22614 leader_election.cc:288] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:47.336408 22759 raft_consensus.cc:482] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:46.876636 22401 raft_consensus.cc:504] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:46.983390 22720 raft_consensus.cc:482] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:47.372556 22285 raft_consensus.cc:482] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:47.334911 22462 raft_consensus.cc:504] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:47.401062 22776 raft_consensus.cc:482] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:47.401485 22462 leader_election.cc:288] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:47.401895 22703 leader_election.cc:288] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:47.401547 22776 raft_consensus.cc:504] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:47.397541 22285 raft_consensus.cc:504] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:47.403788 22776 leader_election.cc:288] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:47.385923 22759 raft_consensus.cc:504] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:47.426630 18189 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50060 (request call id 141) took 1151 ms. Trace:
I20260807 03:39:47.426990 18189 rpcz_store.cc:276] 0807 03:39:46.275227 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:46.647217 (+371990us) service_pool.cc:227] Handling call
0807 03:39:47.425943 (+778726us) spinlock_profiling.cc:243] Waited 11.5 ms on lock 0x7b5800003998. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f25f851 00007fe98f25f702 00007fe99814fb09 00007fe99708b730 00007fe99715179b 00007fe997151821 00007fe9971152b0 00007fe9971007f9 00007fe9971002a2 00007fe9907e6684
0807 03:39:47.426116 (+ 173us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:47.426316 (+ 200us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:47.426338 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:47.426370 (+ 32us) raft_consensus.cc:1631] Marking committed up to 7
0807 03:39:47.426408 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:47.426467 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:47.426578 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":26540416}
I20260807 03:39:47.428227 22759 leader_election.cc:288] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:47.363430 22737 raft_consensus.cc:504] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:47.437840 22763 raft_consensus.cc:482] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5c9b5062bdfa46af9396f1a85025a331)
I20260807 03:39:47.438555 22737 leader_election.cc:288] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:47.438524 22763 raft_consensus.cc:504] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:47.441028 22739 leader_election.cc:288] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:47.443276 22667 raft_consensus.cc:979] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02: : Instructing follower 66a36e1ac8e945e893668e282d394c49 to start an election
I20260807 03:39:47.443810 22715 leader_election.cc:288] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:47.443908 22667 raft_consensus.cc:1067] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Signalling peer 66a36e1ac8e945e893668e282d394c49 to start an election
W20260807 03:39:47.445500 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 528244us
I20260807 03:39:47.404969 22285 leader_election.cc:288] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:47.390929 22774 raft_consensus.cc:482] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:47.362809 22589 raft_consensus.cc:504] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:47.409364 22401 leader_election.cc:288] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:47.385712 22367 raft_consensus.cc:482] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
W20260807 03:39:47.410957 18025 consensus_peers.cc:487] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d -> Peer 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381): Couldn't send request to peer 5c9b5062bdfa46af9396f1a85025a331. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b66966247c147308a4b8488557e0bdf. This is attempt 1: this message will repeat every 5th retry.
I20260807 03:39:47.412834 22740 leader_election.cc:288] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:47.412665 22550 raft_consensus.cc:2781] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:39:47.469307 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:46.890429 22753 raft_consensus.cc:504] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:47.414551 22581 leader_election.cc:288] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:47.472283 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:47.474826 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:47.396555 22720 raft_consensus.cc:504] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:47.473033 18312 raft_consensus.cc:2454] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 8.
I20260807 03:39:47.421553 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:47.424911 22441 leader_election.cc:288] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:47.374629 22686 raft_consensus.cc:504] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:47.392213 22614 raft_consensus.cc:482] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 03:39:47.464164 18025 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.132:41381, user_credentials={real_user=slave}} blocked reactor thread for 603684us
I20260807 03:39:47.373517 22682 raft_consensus.cc:504] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:47.466991 18298 raft_consensus.cc:1261] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260807 03:39:47.376508 22644 raft_consensus.cc:504] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:47.467402 22550 raft_consensus.cc:482] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:47.380541 22602 raft_consensus.cc:504] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:47.380031 22762 raft_consensus.cc:504] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:47.678134 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:47.678856 18312 raft_consensus.cc:2454] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 7.
I20260807 03:39:47.682679 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:47.683755 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:47.684450 18138 leader_election.cc:302] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331; no voters:
I20260807 03:39:47.683912 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:47.687896 22737 raft_consensus.cc:482] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:47.688426 22783 raft_consensus.cc:482] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5c9b5062bdfa46af9396f1a85025a331)
I20260807 03:39:47.689009 22783 raft_consensus.cc:504] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:47.688462 22737 raft_consensus.cc:504] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:47.690588 22769 raft_consensus.cc:482] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:47.691377 22737 leader_election.cc:288] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:47.691123 22769 raft_consensus.cc:504] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:47.466427 22367 raft_consensus.cc:504] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:47.693033 22686 leader_election.cc:288] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:47.693181 22769 leader_election.cc:288] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:47.693475 22367 leader_election.cc:288] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:47.694578 22720 leader_election.cc:288] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:47.694821 22755 raft_consensus.cc:482] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:47.695345 22755 raft_consensus.cc:504] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:47.696563 18069 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33268 (request call id 1913) took 1545 ms (client timeout 1655 ms). Trace:
W20260807 03:39:47.696969 18069 rpcz_store.cc:269] 0807 03:39:46.150953 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:46.151671 (+ 718us) service_pool.cc:227] Handling call
0807 03:39:47.696398 (+1544727us) spinlock_profiling.cc:243] Waited 27.8 ms on lock 0x7b54002c9054. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:47.696503 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":63898496}
I20260807 03:39:47.697378 22755 leader_election.cc:288] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:47.697559 22773 raft_consensus.cc:482] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:47.698041 22755 raft_consensus.cc:482] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:47.698145 22773 raft_consensus.cc:504] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:47.698506 22755 raft_consensus.cc:504] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:47.700225 22686 raft_consensus.cc:482] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5c9b5062bdfa46af9396f1a85025a331)
I20260807 03:39:47.700333 22783 leader_election.cc:288] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:47.700758 22686 raft_consensus.cc:504] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:47.701315 22703 raft_consensus.cc:2781] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Leader election won for term 10
I20260807 03:39:47.701850 22703 raft_consensus.cc:686] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 10 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:47.702580 22753 leader_election.cc:288] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:47.702598 22686 leader_election.cc:288] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:47.702760 22703 consensus_queue.cc:237] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:47.462831 22774 raft_consensus.cc:504] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:47.705478 22774 leader_election.cc:288] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:47.705838 19292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:47.706533 22411 raft_consensus.cc:482] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:47.706997 22411 raft_consensus.cc:504] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:47.708097 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:47.708436 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:47.708720 18075 raft_consensus.cc:2454] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 11.
I20260807 03:39:47.708863 22411 leader_election.cc:288] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:47.709580 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:47.710217 18070 raft_consensus.cc:2454] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 9.
W20260807 03:39:47.710557 22646 log.cc:924] Time spent T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 2.194s user 0.011s sys 0.027s
I20260807 03:39:47.681919 22715 raft_consensus.cc:482] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:47.712230 18309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37190 (request call id 142) took 2766 ms. Trace:
I20260807 03:39:47.712343 22649 raft_consensus.cc:482] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:47.714843 18081 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33268 (request call id 1918) took 1561 ms. Trace:
I20260807 03:39:47.715651 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:47.706100 22754 leader_election.cc:288] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:47.712987 22649 raft_consensus.cc:504] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:47.712086 22715 raft_consensus.cc:504] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:47.684593 18297 raft_consensus.cc:2454] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 8.
I20260807 03:39:47.866796 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:47.697574 18300 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37190 (request call id 147) took 1409 ms. Trace:
I20260807 03:39:47.715302 18081 rpcz_store.cc:276] 0807 03:39:46.153544 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:46.158106 (+ 4562us) service_pool.cc:227] Handling call
0807 03:39:46.158424 (+ 318us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:46.285442 (+127018us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:46.285469 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:46.285502 (+ 33us) raft_consensus.cc:1631] Marking committed up to 4
0807 03:39:46.285539 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:47.714562 (+1429023us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x7b540005a854. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:39:47.714626 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:47.714774 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":291675520}
I20260807 03:39:47.907998 18300 rpcz_store.cc:276] 0807 03:39:46.289255 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:47.696853 (+1407598us) service_pool.cc:227] Handling call
0807 03:39:47.697129 (+ 276us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:47.697288 (+ 159us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:47.697311 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:47.697345 (+ 34us) raft_consensus.cc:1631] Marking committed up to 5
0807 03:39:47.697386 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:47.697432 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:47.697518 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 03:39:47.716269 18071 raft_consensus.cc:2454] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 8.
I20260807 03:39:47.716931 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:47.737862 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 50698.8us
I20260807 03:39:47.850067 18027 leader_election.cc:302] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:47.888969 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:47.913758 18184 raft_consensus.cc:2454] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:47.821341 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:47.915593 18189 raft_consensus.cc:2454] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 12.
I20260807 03:39:47.717324 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:47.712787 18309 rpcz_store.cc:276] 0807 03:39:44.945887 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:44.946044 (+ 157us) service_pool.cc:227] Handling call
0807 03:39:44.946373 (+ 329us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:44.958644 (+ 12271us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:44.958670 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:39:44.960181 (+ 1511us) log.cc:842] Serialized 26 byte log entry
0807 03:39:45.216085 (+255904us) raft_consensus.cc:1631] Marking committed up to 6
0807 03:39:45.242504 (+ 26419us) log.cc:842] Serialized 14 byte log entry
0807 03:39:47.436913 (+2194409us) spinlock_profiling.cc:243] Waited 44.2 ms on lock 0x7b180034de00. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe9981368b1 00007fe998143661 00007fe991e4e82c 00007fe98f2799dc 00007fe9980b0a39 00007fe98f2968bb 00007fe98f294cbf 00007fe98f294155 00007fe9921b73d0 00007fe9921c9576 00007fe9921da0d7
0807 03:39:47.437055 (+ 142us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:47.437154 (+ 99us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:47.691244 (+254090us) spinlock_profiling.cc:243] Waited 4.03 ms on lock 0x7b180034de00. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe9981368b1 00007fe998143661 00007fe98f297764 00007fe98f29ab11 00007fe98f29aac9 00007fe98f29aa59 00007fe98f29aa21 00007fe98f299d1d 00007fe9984fb204 00007fe9984fb039 00007fe98f279dc6
0807 03:39:47.709662 (+ 18418us) spinlock_profiling.cc:243] Waited 7.76 ms on lock 0x7b44000232cc. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f1a9a95 00007fe98f1aa4b2 00007fe991e52511 00007fe991e4de1a 00007fe991e41bee 00007fe991e40f36 00007fe991e40b58 00007fe991e4a551 00007fe991e4a509 00007fe991e4a499
0807 03:39:47.712026 (+ 2364us) raft_consensus.cc:1671] finished
0807 03:39:47.712046 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:47.712174 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":59297,"prepare.queue_time_us":19322,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":57,"raft.queue_time_us":157413,"raft.run_cpu_time_us":297,"raft.run_wall_time_us":26930,"spinlock_wait_cycles":128712448,"thread_start_us":346100,"threads_started":2,"wal-append.queue_time_us":400831}
I20260807 03:39:47.837834 18082 raft_consensus.cc:2354] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 7. Current term is 8.
I20260807 03:39:47.916666 18069 raft_consensus.cc:2454] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 6.
I20260807 03:39:47.910382 18141 leader_election.cc:302] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:47.917312 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:47.845414 18311 raft_consensus.cc:2454] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 7.
I20260807 03:39:47.934886 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:47.929728 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:47.904606 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:47.803651 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:47.889276 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:47.688738 22660 raft_consensus.cc:482] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:47.909461 18295 raft_consensus.cc:2454] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 6.
W20260807 03:39:47.887717 22676 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 1.615s user 0.010s sys 0.019s
W20260807 03:39:47.948966 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 168131us
I20260807 03:39:47.951776 19297 raft_consensus.cc:1226] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [term 10 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 9. Current term is 10. Ops: []
I20260807 03:39:47.559926 22441 raft_consensus.cc:482] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:47.510149 22787 raft_consensus.cc:482] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:47.886276 22777 raft_consensus.cc:482] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:47.643759 22667 raft_consensus.cc:482] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:47.685580 22550 raft_consensus.cc:3037] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:47.954448 22790 raft_consensus.cc:482] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:47.886468 22678 raft_consensus.cc:482] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:47.658641 22759 raft_consensus.cc:482] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:47.712280 22755 leader_election.cc:288] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:47.803160 22530 raft_consensus.cc:482] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
W20260807 03:39:47.959434 22647 log.cc:924] Time spent T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 2.073s user 0.004s sys 0.034s
I20260807 03:39:47.959714 22720 raft_consensus.cc:482] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 5c9b5062bdfa46af9396f1a85025a331)
I20260807 03:39:47.496723 22763 leader_election.cc:288] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:47.945917 22774 raft_consensus.cc:2781] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:39:47.960000 22769 raft_consensus.cc:482] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:47.960520 22789 consensus_queue.cc:1035] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 03:39:47.945014 22740 raft_consensus.cc:482] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:47.961324 18188 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44662 (request call id 1600) took 1461 ms. Trace:
I20260807 03:39:47.961481 22715 leader_election.cc:288] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:47.962411 18180 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44662 (request call id 1601) took 1461 ms. Trace:
W20260807 03:39:47.949983 22676 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 2.054s user 0.012s sys 0.022s
I20260807 03:39:47.946895 22776 raft_consensus.cc:482] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:47.965988 22658 raft_consensus.cc:979] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02: : Instructing follower 66a36e1ac8e945e893668e282d394c49 to start an election
I20260807 03:39:47.968482 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:47.963807 18191 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50060 (request call id 142) took 1678 ms. Trace:
I20260807 03:39:47.969794 19297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:47.682137 22589 leader_election.cc:288] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:47.952626 19295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:47.976500 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:47.978155 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:47.661180 22401 raft_consensus.cc:482] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:47.877760 22773 leader_election.cc:288] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:47.848469 22411 raft_consensus.cc:482] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:48.390846 22411 raft_consensus.cc:504] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:48.391961 22763 raft_consensus.cc:482] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:48.392542 22763 raft_consensus.cc:504] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:48.392970 22411 leader_election.cc:288] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:47.984050 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:47.887482 22741 raft_consensus.cc:482] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:47.987787 18190 raft_consensus.cc:1261] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 03:39:47.917639 22462 raft_consensus.cc:482] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:47.848210 22285 raft_consensus.cc:482] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5c9b5062bdfa46af9396f1a85025a331)
I20260807 03:39:48.394795 22462 raft_consensus.cc:504] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:47.961555 19290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:48.012203 22790 raft_consensus.cc:504] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:48.012557 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:47.953001 22754 raft_consensus.cc:2781] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:39:48.397660 18080 raft_consensus.cc:2454] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 6.
I20260807 03:39:47.970912 22644 leader_election.cc:288] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:47.760483 22614 raft_consensus.cc:504] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:47.989144 22705 raft_consensus.cc:482] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:47.989468 22777 raft_consensus.cc:504] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:48.468077 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:47.980238 22441 raft_consensus.cc:504] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:47.991539 22686 consensus_queue.cc:1046] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.096s
I20260807 03:39:48.484638 19289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:48.485152 22441 leader_election.cc:288] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:48.464556 22750 raft_consensus.cc:482] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:47.936800 22549 raft_consensus.cc:482] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:47.953825 22660 raft_consensus.cc:504] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:48.190019 22759 raft_consensus.cc:504] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:48.489755 22614 leader_election.cc:288] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:48.479516 22705 raft_consensus.cc:504] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:48.491304 22759 leader_election.cc:288] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:48.190889 22678 raft_consensus.cc:504] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:48.489796 22549 raft_consensus.cc:504] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:48.493062 22705 leader_election.cc:288] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:48.493505 22678 leader_election.cc:288] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:48.197439 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 347414us
I20260807 03:39:48.494086 22549 leader_election.cc:288] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:48.495515 22777 leader_election.cc:288] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:48.495935 19290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:47.937009 22669 raft_consensus.cc:482] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:48.500370 22669 raft_consensus.cc:504] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:48.198499 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 207641us
I20260807 03:39:48.200795 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:48.201399 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:48.203274 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:48.206694 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:48.206928 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:48.205453 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:48.209806 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:47.993469 22667 raft_consensus.cc:504] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:48.052157 22787 raft_consensus.cc:504] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:48.266346 22720 raft_consensus.cc:504] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:48.406430 22754 raft_consensus.cc:482] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:48.305461 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:48.518828 18305 raft_consensus.cc:2454] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 5.
I20260807 03:39:48.400923 19297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:48.397650 18191 rpcz_store.cc:276] 0807 03:39:46.285612 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:46.285772 (+ 160us) service_pool.cc:227] Handling call
0807 03:39:47.963067 (+1677295us) spinlock_profiling.cc:243] Waited 9.89 ms on lock 0x7b5800003998. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f25f851 00007fe98f25f702 00007fe99814fb09 00007fe99708b730 00007fe99715179b 00007fe997151821 00007fe9971152b0 00007fe9971007f9 00007fe9971002a2 00007fe9907e6684
0807 03:39:47.963291 (+ 224us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:47.963513 (+ 222us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:47.963538 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:47.963588 (+ 50us) raft_consensus.cc:1631] Marking committed up to 6
0807 03:39:47.963628 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:47.963674 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:47.963754 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":22752512}
I20260807 03:39:48.405382 18185 raft_consensus.cc:2454] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 7.
I20260807 03:39:47.849350 22288 consensus_queue.cc:1035] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.132s
I20260807 03:39:48.351497 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:48.286406 18141 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.15.198.131:33021 timed out after 1.514s (SENT) [suppressed 11 similar messages]
I20260807 03:39:48.400843 18180 rpcz_store.cc:276] 0807 03:39:46.500697 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:46.503343 (+ 2646us) service_pool.cc:227] Handling call
0807 03:39:46.503777 (+ 434us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:46.679783 (+176006us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:46.679826 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:39:46.691971 (+ 12145us) log.cc:842] Serialized 26 byte log entry
0807 03:39:46.761949 (+ 69978us) raft_consensus.cc:1631] Marking committed up to 3
0807 03:39:46.761988 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:47.961969 (+1199981us) spinlock_profiling.cc:243] Waited 176 ms on lock 0x7b54002a2e54. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:39:47.962013 (+ 44us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:47.962066 (+ 53us) raft_consensus.cc:1671] finished
0807 03:39:47.962079 (+ 13us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:47.962313 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":76703,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":11758,"spinlock_wait_cycles":403716992,"thread_start_us":67211,"threads_started":1,"wal-append.queue_time_us":185436,"wal-append.run_cpu_time_us":1729,"wal-append.run_wall_time_us":1035011}
I20260807 03:39:48.399111 18188 rpcz_store.cc:276] 0807 03:39:46.500376 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:46.648294 (+147918us) service_pool.cc:227] Handling call
0807 03:39:47.960553 (+1312259us) spinlock_profiling.cc:243] Waited 11.7 ms on lock 0x7b5800003998. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f25f851 00007fe98f25f702 00007fe99814fb09 00007fe99708b730 00007fe99715179b 00007fe997151821 00007fe9971152b0 00007fe9971007f9 00007fe9971002a2 00007fe9907e6684
0807 03:39:47.960772 (+ 219us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:47.960956 (+ 184us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:47.960982 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:47.961038 (+ 56us) raft_consensus.cc:1631] Marking committed up to 5
0807 03:39:47.961080 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:47.961151 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:47.961247 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":26996992}
I20260807 03:39:48.399801 22774 raft_consensus.cc:482] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 03:39:48.096678 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 139803us
I20260807 03:39:48.411252 22602 leader_election.cc:288] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:48.488574 18183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44660 (request call id 2354) took 3532 ms. Trace:
I20260807 03:39:48.491389 22660 leader_election.cc:288] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:48.529323 18183 rpcz_store.cc:276] 0807 03:39:44.947823 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:44.964591 (+ 16768us) service_pool.cc:227] Handling call
0807 03:39:44.964982 (+ 391us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:44.965379 (+ 397us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:44.965412 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:39:44.981340 (+ 15928us) log.cc:842] Serialized 26 byte log entry
0807 03:39:45.217495 (+236155us) raft_consensus.cc:1631] Marking committed up to 3
0807 03:39:45.217542 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:45.217652 (+ 110us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:47.958543 (+2740891us) spinlock_profiling.cc:243] Waited 18.6 ms on lock 0x7b44000232cc. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f1a9a95 00007fe98f1aa4b2 00007fe991e52511 00007fe991e4de1a 00007fe991e41bee 00007fe991e40f36 00007fe991e40b58 00007fe991e4a551 00007fe991e4a509 00007fe991e4a499
0807 03:39:48.464268 (+505725us) raft_consensus.cc:1671] finished
0807 03:39:48.464288 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:48.479972 (+ 15684us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6108,"raft.queue_time_us":1816,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":10925,"spinlock_wait_cycles":42683648,"thread_start_us":217906,"threads_started":1,"wal-append.queue_time_us":641279}
I20260807 03:39:48.503168 18295 raft_consensus.cc:2454] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 12.
I20260807 03:39:48.205127 18300 raft_consensus.cc:1261] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260807 03:39:48.504384 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:48.533067 18297 raft_consensus.cc:2454] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 7.
I20260807 03:39:48.533747 18297 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c98c1d422654739af06653a512f7499"
dest_uuid: "66a36e1ac8e945e893668e282d394c49"
from {username='slave'} at 127.0.0.1:57268
W20260807 03:39:48.209920 18310 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57276 (request call id 1654) took 1533 ms (client timeout 1698 ms). Trace:
I20260807 03:39:48.212158 22657 raft_consensus.cc:482] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:48.370088 22762 leader_election.cc:288] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:48.265089 19291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:48.507303 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:48.507951 19240 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.563s user 0.020s sys 0.054s
I20260807 03:39:48.508478 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:48.543350 19240 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.601s user 0.020s sys 0.054s
I20260807 03:39:48.395825 22763 leader_election.cc:288] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:48.266134 22683 raft_consensus.cc:482] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:48.258651 22682 leader_election.cc:288] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:48.394598 22741 raft_consensus.cc:504] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:48.395403 22285 raft_consensus.cc:504] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:48.566247 22285 leader_election.cc:288] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:48.412528 22715 raft_consensus.cc:1067] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Signalling peer 66a36e1ac8e945e893668e282d394c49 to start an election
I20260807 03:39:48.211462 22530 raft_consensus.cc:504] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:48.422021 22401 raft_consensus.cc:504] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:48.540179 19292 thread.cc:636] tablet-copy [worker] (thread pool) Time spent creating pthread: real 0.522s user 0.020s sys 0.032s
I20260807 03:39:48.568291 22741 leader_election.cc:288] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
W20260807 03:39:48.543682 19240 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 604763us
I20260807 03:39:48.569959 22401 leader_election.cc:288] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:48.521339 22754 raft_consensus.cc:3037] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:48.529381 22774 raft_consensus.cc:3037] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:48.461632 22740 raft_consensus.cc:504] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:48.461416 22769 raft_consensus.cc:504] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:48.465576 22776 raft_consensus.cc:504] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:48.513252 22750 raft_consensus.cc:504] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:48.575743 22649 leader_election.cc:288] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:48.543980 22657 raft_consensus.cc:504] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:48.625722 22787 leader_election.cc:288] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:48.626938 22720 leader_election.cc:288] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:48.625667 22790 leader_election.cc:288] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:48.627262 22740 leader_election.cc:288] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:48.629649 22769 leader_election.cc:288] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:48.631027 22667 leader_election.cc:288] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:48.623869 22814 ts_tablet_manager.cc:927] T 6b66966247c147308a4b8488557e0bdf P 5c9b5062bdfa46af9396f1a85025a331: Initiating tablet copy from peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163)
I20260807 03:39:48.573575 22683 raft_consensus.cc:504] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:48.634341 22683 leader_election.cc:288] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:48.634334 22754 raft_consensus.cc:504] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:48.622136 22669 leader_election.cc:288] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:48.636819 22754 leader_election.cc:288] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:48.637575 22774 raft_consensus.cc:504] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:48.638150 22657 leader_election.cc:288] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:48.640231 22550 consensus_meta.cc:287] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.635s user 0.008s sys 0.000s
I20260807 03:39:48.640300 22774 leader_election.cc:288] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:48.640599 22799 log.cc:924] Time spent T 00000000000000000000000000000000 P 8da2065a77024f35bdfc0dc3adb83f69: Append to log took a long time: real 0.135s user 0.000s sys 0.000s
I20260807 03:39:48.642772 22801 raft_consensus.cc:3032] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 9 LEADER]: Stepping down as leader of term 9
I20260807 03:39:48.643051 22750 leader_election.cc:288] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:48.643177 22801 raft_consensus.cc:729] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:48.645267 22776 leader_election.cc:288] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:48.644052 22801 consensus_queue.cc:260] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:48.659483 22801 raft_consensus.cc:3037] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Advancing to term 10
I20260807 03:39:48.503888 18071 raft_consensus.cc:2454] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:39:48.694401 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:48.696310 18071 raft_consensus.cc:2454] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 6.
W20260807 03:39:48.697661 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 202631us
I20260807 03:39:48.423564 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:48.699137 17918 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39492 (request call id 221) took 3695 ms. Trace:
I20260807 03:39:48.462265 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: true dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:48.700400 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ca0223f47b46fd955704eba5259370" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:48.701370 18072 raft_consensus.cc:2454] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 2.
I20260807 03:39:48.702780 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:48.704442 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:48.705601 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:39:48.479157 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:48.712463 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:48.714417 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:48.305869 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:48.717381 18082 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33268 (request call id 1912) took 2566 ms (client timeout 1903 ms). Trace:
I20260807 03:39:48.699661 17918 rpcz_store.cc:276] 0807 03:39:45.003418 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:45.003589 (+ 171us) service_pool.cc:227] Handling call
0807 03:39:48.698783 (+3695194us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 03:39:45.200010 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 03:39:45.200612 (+ 602us) write_op.cc:432] Acquiring schema lock in shared mode
0807 03:39:45.200653 (+ 41us) write_op.cc:435] Acquired schema lock
0807 03:39:45.200671 (+ 18us) tablet.cc:660] Decoding operations
0807 03:39:45.201027 (+ 356us) write_op.cc:620] Acquiring the partition lock for write op
0807 03:39:45.201084 (+ 57us) write_op.cc:641] Partition lock acquired for write op
0807 03:39:45.201111 (+ 27us) tablet.cc:683] Acquiring locks for 1 operations
0807 03:39:45.201296 (+ 185us) tablet.cc:699] Row locks acquired
0807 03:39:45.201310 (+ 14us) write_op.cc:260] PREPARE: finished
0807 03:39:45.201448 (+ 138us) write_op.cc:270] Start()
0807 03:39:45.201570 (+ 122us) write_op.cc:276] Timestamp: P: 1786073985201406 usec, L: 0
0807 03:39:45.201587 (+ 17us) op_driver.cc:331] REPLICATION: starting
0807 03:39:45.202390 (+ 803us) log.cc:842] Serialized 544 byte log entry
0807 03:39:45.895322 (+692932us) op_driver.cc:444] REPLICATION: finished
0807 03:39:47.438000 (+1542678us) spinlock_profiling.cc:243] Waited 15.6 ms on lock 0x7b44000232cc. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f1a9a95 00007fe98f1aa4b2 00007fe991e52511 00007fe991e4de1a 00007fe98f2795fe 00007fe9980b0a39 00007fe98f2968bb 00007fe98f294cbf 00007fe98f296e22 00007fe995bcf410
0807 03:39:47.847255 (+409255us) write_op.cc:301] APPLY: starting
0807 03:39:47.847368 (+ 113us) tablet.cc:1365] starting BulkCheckPresence
0807 03:39:47.847501 (+ 133us) tablet.cc:1368] finished BulkCheckPresence
0807 03:39:47.847530 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 03:39:47.847843 (+ 313us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 03:39:47.849536 (+ 1693us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0807 03:39:47.849603 (+ 67us) write_op.cc:312] APPLY: finished
0807 03:39:47.849849 (+ 246us) log.cc:842] Serialized 23 byte log entry
0807 03:39:48.085559 (+235710us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:39:48.088556 (+ 2997us) write_op.cc:454] Released schema lock
0807 03:39:48.097079 (+ 8523us) write_op.cc:341] FINISH: Updating metrics
0807 03:39:48.196427 (+ 99348us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:39:48.199833 (+ 3406us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":2,"child_traces":[["op",{"apply.queue_time_us":1951615,"apply.run_cpu_time_us":35396,"apply.run_wall_time_us":411639,"num_ops":1,"prepare.queue_time_us":45526,"prepare.run_cpu_time_us":2672,"prepare.run_wall_time_us":2672,"replication_time_us":693613,"spinlock_wait_cycles":35934592,"thread_start_us":2689308,"threads_started":2,"wal-append.queue_time_us":519305}]]}
W20260807 03:39:48.717917 18082 rpcz_store.cc:269] 0807 03:39:46.150622 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:46.150772 (+ 150us) service_pool.cc:227] Handling call
0807 03:39:48.716879 (+2566107us) spinlock_profiling.cc:243] Waited 1.66 s on lock 0x7b540037da54. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:48.717137 (+ 258us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3820179328}
I20260807 03:39:48.720597 18027 leader_election.cc:302] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:48.719780 17915 catalog_manager.cc:5495] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:48.526017 18140 leader_election.cc:302] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:48.724803 18072 raft_consensus.cc:1226] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 6. Current term is 7. Ops: []
I20260807 03:39:48.775974 22530 leader_election.cc:288] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:48.524312 18193 raft_consensus.cc:2454] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 6.
I20260807 03:39:48.777397 19290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:48.778236 18193 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50060 (request call id 146) took 1009 ms. Trace:
I20260807 03:39:48.778690 18193 rpcz_store.cc:276] 0807 03:39:47.769210 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:47.769342 (+ 132us) service_pool.cc:227] Handling call
0807 03:39:48.777769 (+1008427us) spinlock_profiling.cc:243] Waited 588 ms on lock 0x7b5400194954. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:48.778014 (+ 245us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1352360704}
I20260807 03:39:48.640990 22550 raft_consensus.cc:504] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:48.517333 18248 leader_election.cc:302] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:48.534209 18297 raft_consensus.cc:482] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:39:48.781518 18297 raft_consensus.cc:3037] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:48.782564 22550 leader_election.cc:288] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163)
I20260807 03:39:48.782742 19289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:48.538424 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:48.740653 18298 raft_consensus.cc:2454] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:39:48.538316 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ca0223f47b46fd955704eba5259370" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:48.534571 18310 rpcz_store.cc:269] 0807 03:39:46.676145 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:46.676314 (+ 169us) service_pool.cc:227] Handling call
0807 03:39:48.209705 (+1533391us) spinlock_profiling.cc:243] Waited 983 ms on lock 0x7b54001c3254. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:48.209831 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2260571904}
I20260807 03:39:48.543597 18309 raft_consensus.cc:2454] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 5.
W20260807 03:39:48.516815 18311 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57268 (request call id 1992) took 2349 ms (client timeout 1514 ms). Trace:
W20260807 03:39:48.568467 19292 thread.cc:602] tablet-copy [worker] (thread pool) Time spent starting thread: real 0.554s user 0.020s sys 0.032s
I20260807 03:39:48.745122 19297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
W20260807 03:39:48.805580 18311 rpcz_store.cc:269] 0807 03:39:46.167873 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:46.170884 (+ 3011us) service_pool.cc:227] Handling call
0807 03:39:48.516555 (+2345671us) spinlock_profiling.cc:243] Waited 1.67 s on lock 0x7b54001d0e54. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:48.516711 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3837776256}
I20260807 03:39:48.697820 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:48.806699 18070 raft_consensus.cc:1261] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 03:39:48.767834 22462 leader_election.cc:288] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:48.807266 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 279008us
I20260807 03:39:48.807440 18070 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33254 (request call id 2243) took 1092 ms. Trace:
I20260807 03:39:48.770457 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:48.788383 18312 raft_consensus.cc:2379] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 7: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:48.810149 18140 leader_election.cc:302] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:48.810549 22788 consensus_queue.cc:1035] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260807 03:39:48.797129 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 274851us
I20260807 03:39:48.802840 22532 raft_consensus.cc:2781] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260807 03:39:48.807929 18070 rpcz_store.cc:276] 0807 03:39:47.714993 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:47.724317 (+ 9324us) service_pool.cc:227] Handling call
0807 03:39:48.707836 (+983519us) spinlock_profiling.cc:243] Waited 17.4 ms on lock 0x7b5800001298. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f25fc6c 00007fe98f25fbd6 000000000034aea9 00007fe99708ba30 00007fe9971517b4 00007fe997151821 00007fe9971152b0 00007fe9971007f9 00007fe9971002a2 00007fe9907e6684
0807 03:39:48.722434 (+ 14598us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:48.807122 (+ 84688us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:48.807351 (+ 229us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":40059904}
I20260807 03:39:48.810755 22658 consensus_queue.cc:1035] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260807 03:39:48.801187 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 80873.5us
I20260807 03:39:48.801645 22476 raft_consensus.cc:2781] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:39:48.814074 18248 leader_election.cc:302] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:48.814690 22476 raft_consensus.cc:482] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:48.815136 22476 raft_consensus.cc:3037] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:48.816967 22817 raft_consensus.cc:2726] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:48.816996 22532 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:48.822435 22532 raft_consensus.cc:3037] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 12 FOLLOWER]: Advancing to term 13
I20260807 03:39:48.811832 22774 raft_consensus.cc:2781] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 03:39:48.825062 22774 raft_consensus.cc:482] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:48.825507 22774 raft_consensus.cc:3037] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:48.828680 19240 leader_election.cc:302] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331; no voters:
I20260807 03:39:48.830174 22581 raft_consensus.cc:2781] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:39:48.830735 22581 raft_consensus.cc:482] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:48.830888 18027 leader_election.cc:302] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:48.831182 22581 raft_consensus.cc:3037] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:48.832183 22462 raft_consensus.cc:2781] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 03:39:48.832561 22462 raft_consensus.cc:482] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:48.832875 22462 raft_consensus.cc:3037] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:48.830111 22739 raft_consensus.cc:2781] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 03:39:48.836449 22739 raft_consensus.cc:482] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:48.837028 22739 raft_consensus.cc:3037] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:48.811904 18140 leader_election.cc:302] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:48.854133 18027 leader_election.cc:302] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:48.858341 18069 raft_consensus.cc:2454] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 10.
I20260807 03:39:48.858397 22132 consensus_queue.cc:1035] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.002s
W20260807 03:39:48.861018 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 298608us
I20260807 03:39:48.862031 22288 raft_consensus.cc:2781] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 03:39:48.862483 22288 raft_consensus.cc:482] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:48.862879 22288 raft_consensus.cc:3037] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:48.866166 18027 leader_election.cc:302] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:48.867573 22814 tablet_copy_client.cc:316] T 6b66966247c147308a4b8488557e0bdf P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Beginning tablet copy session from remote peer at address 127.15.198.129:36163
I20260807 03:39:48.868366 18248 leader_election.cc:302] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:48.870137 22776 raft_consensus.cc:2781] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260807 03:39:48.870559 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 58962.9us
I20260807 03:39:48.872021 22658 raft_consensus.cc:2781] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260807 03:39:48.876230 22776 raft_consensus.cc:482] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:48.876637 22658 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:48.876873 18248 leader_election.cc:302] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:48.876754 22776 raft_consensus.cc:3037] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:48.877189 22658 raft_consensus.cc:3037] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 12 FOLLOWER]: Advancing to term 13
I20260807 03:39:48.879020 22762 raft_consensus.cc:2781] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260807 03:39:48.879484 22762 raft_consensus.cc:482] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:48.879910 22762 raft_consensus.cc:3037] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Advancing to term 10
I20260807 03:39:48.862798 18249 leader_election.cc:302] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:39:48.884778 22817 raft_consensus.cc:2781] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:39:48.885475 22817 raft_consensus.cc:482] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 5c9b5062bdfa46af9396f1a85025a331)
I20260807 03:39:48.886009 22817 raft_consensus.cc:3037] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:48.878177 18248 leader_election.cc:302] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:48.884869 22748 raft_consensus.cc:2781] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 03:39:48.890806 18297 raft_consensus.cc:504] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:48.893164 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 586769us
I20260807 03:39:48.893379 22748 raft_consensus.cc:482] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:48.896955 22520 raft_consensus.cc:2781] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:39:48.900312 22748 raft_consensus.cc:3037] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:48.900393 22520 raft_consensus.cc:482] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:48.900928 22520 raft_consensus.cc:3037] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:48.907317 18297 leader_election.cc:288] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:48.912833 22530 raft_consensus.cc:482] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:48.913582 22530 raft_consensus.cc:504] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:48.906586 22462 raft_consensus.cc:504] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:48.916864 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:48.917739 18293 raft_consensus.cc:2454] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:39:48.918639 22462 leader_election.cc:288] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:48.920415 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:48.921272 18293 raft_consensus.cc:2454] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 7.
I20260807 03:39:48.923907 18248 leader_election.cc:302] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:48.925119 18094 tablet_copy_service.cc:140] P d9716e2bf06442f5b3b0e81087c53c4d: Received BeginTabletCopySession request for tablet 6b66966247c147308a4b8488557e0bdf from peer 5c9b5062bdfa46af9396f1a85025a331 ({username='slave'} at 127.0.0.1:58950)
I20260807 03:39:48.925128 18293 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bb66518c65e4495e881ce860253d0801"
dest_uuid: "66a36e1ac8e945e893668e282d394c49"
from {username='slave'} at 127.0.0.1:57268
I20260807 03:39:48.925931 18094 tablet_copy_service.cc:161] P d9716e2bf06442f5b3b0e81087c53c4d: Beginning new tablet copy session on tablet 6b66966247c147308a4b8488557e0bdf from peer 5c9b5062bdfa46af9396f1a85025a331 at {username='slave'} at 127.0.0.1:58950: session id = 5c9b5062bdfa46af9396f1a85025a331-6b66966247c147308a4b8488557e0bdf
I20260807 03:39:48.926496 18293 raft_consensus.cc:482] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:39:48.926697 22530 leader_election.cc:288] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:48.926995 18293 raft_consensus.cc:3037] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:48.904049 22776 raft_consensus.cc:504] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:48.928480 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:48.931113 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:48.931946 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:48.932857 18310 raft_consensus.cc:3037] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:48.931949 18311 raft_consensus.cc:2454] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 5.
I20260807 03:39:48.934544 18140 leader_election.cc:302] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:48.933030 22774 raft_consensus.cc:504] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:48.936704 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:48.942128 22774 leader_election.cc:288] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:48.938078 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:48.953255 18298 raft_consensus.cc:2454] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 7.
I20260807 03:39:48.953392 22520 raft_consensus.cc:504] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:48.945013 22776 leader_election.cc:288] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:48.945201 22288 raft_consensus.cc:504] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:48.944271 22476 raft_consensus.cc:504] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:48.950695 18310 raft_consensus.cc:2454] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 6.
I20260807 03:39:48.951308 22532 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:48.943179 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:48.936868 22581 raft_consensus.cc:504] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:48.958297 18293 raft_consensus.cc:504] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:48.961580 22550 raft_consensus.cc:2726] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 03:39:48.962095 18140 leader_election.cc:302] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:48.962616 22288 leader_election.cc:288] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:48.963359 22760 raft_consensus.cc:482] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:48.963384 22257 raft_consensus.cc:2781] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:48.963546 22476 leader_election.cc:288] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:48.963997 22257 raft_consensus.cc:482] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:48.963953 22760 raft_consensus.cc:504] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:48.964572 22257 raft_consensus.cc:3037] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:48.968945 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:48.971539 18309 raft_consensus.cc:1226] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260807 03:39:48.972681 18310 raft_consensus.cc:2454] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 4.
I20260807 03:39:48.975378 22760 leader_election.cc:288] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:48.969889 18094 tablet_copy_source_session.cc:205] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 03:39:48.976337 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:48.977056 18069 raft_consensus.cc:2454] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 10.
I20260807 03:39:48.978607 18249 leader_election.cc:302] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:48.978885 19239 leader_election.cc:302] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:48.979633 22520 leader_election.cc:288] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
W20260807 03:39:48.980238 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 56523.1us
I20260807 03:39:48.980291 22760 raft_consensus.cc:2781] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260807 03:39:48.980952 22760 raft_consensus.cc:482] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:48.981101 18248 leader_election.cc:302] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:48.981436 22760 raft_consensus.cc:3037] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [term 10 FOLLOWER]: Advancing to term 11
I20260807 03:39:48.982918 22532 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 13 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:48.986014 22427 raft_consensus.cc:2726] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 03:39:48.955343 22817 raft_consensus.cc:504] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:48.992452 22520 consensus_queue.cc:1035] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 03:39:48.994451 22817 leader_election.cc:288] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:48.995716 19295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331"
I20260807 03:39:48.990535 18249 leader_election.cc:302] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:48.996882 22581 leader_election.cc:288] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 7 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:49.000375 22530 raft_consensus.cc:2781] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:39:48.998986 18293 leader_election.cc:288] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:49.009785 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:49.003433 22530 raft_consensus.cc:482] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:49.013358 22530 raft_consensus.cc:3037] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:49.014397 22657 raft_consensus.cc:482] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:49.015027 22657 raft_consensus.cc:504] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:48.965045 22739 raft_consensus.cc:504] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:49.017802 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:49.003583 22520 raft_consensus.cc:2781] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 03:39:49.019315 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:49.006047 22817 raft_consensus.cc:2741] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260807 03:39:49.021772 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:49.036466 18140 leader_election.cc:302] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:49.036720 18311 raft_consensus.cc:2454] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 4.
I20260807 03:39:49.036743 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:49.022893 18298 raft_consensus.cc:2454] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 2.
I20260807 03:39:49.037789 18188 raft_consensus.cc:2454] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 7.
I20260807 03:39:49.038842 22476 raft_consensus.cc:2781] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:39:49.039378 22476 raft_consensus.cc:482] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:49.039863 22476 raft_consensus.cc:3037] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:49.040993 18140 leader_election.cc:302] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:49.023417 22657 leader_election.cc:288] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:49.023118 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:49.023195 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:49.024333 22762 raft_consensus.cc:504] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:49.045018 22520 raft_consensus.cc:686] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:49.046638 22762 leader_election.cc:288] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:49.046475 22520 consensus_queue.cc:237] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:49.044324 18141 leader_election.cc:302] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:49.048138 22788 raft_consensus.cc:2726] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 03:39:49.049458 18141 leader_election.cc:302] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:49.050307 22788 raft_consensus.cc:2726] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 03:39:49.051620 18141 leader_election.cc:302] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:49.053488 22257 raft_consensus.cc:504] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:49.055480 22257 leader_election.cc:288] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:49.029548 22739 leader_election.cc:288] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:49.035418 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:49.056021 18069 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1f5db05c008b404288067417c3f100d2"
dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
mode: GRACEFUL
new_leader_uuid: "3c4bceb836db4536a0f255fdf86abc02"
from {username='slave'} at 127.0.0.1:55742
I20260807 03:39:49.056469 18193 raft_consensus.cc:2454] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:49.056734 18069 raft_consensus.cc:595] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Received request to transfer leadership to TS 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:49.060420 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:49.060813 22814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b66966247c147308a4b8488557e0bdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:39:49.061178 18298 raft_consensus.cc:2454] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 9.
I20260807 03:39:49.063422 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:49.063999 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:49.064261 18310 raft_consensus.cc:2454] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 6.
I20260807 03:39:49.064397 18193 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
mode: GRACEFUL
new_leader_uuid: "66a36e1ac8e945e893668e282d394c49"
from {username='slave'} at 127.0.0.1:34758
I20260807 03:39:49.064733 18295 raft_consensus.cc:2454] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 3.
I20260807 03:39:49.064962 18193 raft_consensus.cc:595] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Received request to transfer leadership to TS 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:49.062872 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:49.072407 18193 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
mode: GRACEFUL
new_leader_uuid: "66a36e1ac8e945e893668e282d394c49"
from {username='slave'} at 127.0.0.1:34758
I20260807 03:39:49.073446 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:49.074177 18188 raft_consensus.cc:2454] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 7.
I20260807 03:39:49.074712 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:49.076202 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:49.076881 18183 raft_consensus.cc:2454] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
I20260807 03:39:49.079363 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:49.064848 22817 consensus_queue.cc:1035] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:39:49.082955 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
W20260807 03:39:49.083884 19240 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 115048us
I20260807 03:39:49.094389 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:49.012781 22658 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:49.094503 18298 raft_consensus.cc:3037] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Advancing to term 8
W20260807 03:39:49.095314 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 43793.5us
I20260807 03:39:49.100306 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:49.101843 18141 leader_election.cc:302] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:49.103124 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:49.103120 18193 raft_consensus.cc:595] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 13 FOLLOWER]: Received request to transfer leadership to TS 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:49.101703 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:49.103832 18193 raft_consensus.cc:603] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 03:39:49.102825 22441 raft_consensus.cc:2726] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 03:39:49.090649 22476 raft_consensus.cc:504] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:49.107788 22788 raft_consensus.cc:2781] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:39:49.108217 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:49.108892 18069 raft_consensus.cc:2354] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 7. Current term is 8.
I20260807 03:39:49.090282 22748 raft_consensus.cc:504] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:49.102995 22257 raft_consensus.cc:2726] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260807 03:39:49.121178 18312 raft_consensus.cc:1226] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 8. Current term is 9. Ops: []
I20260807 03:39:49.110675 22788 raft_consensus.cc:482] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:49.121342 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:49.118556 18180 raft_consensus.cc:2454] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 8.
I20260807 03:39:49.122067 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:49.122990 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 82148.9us
I20260807 03:39:49.131629 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:49.125871 22788 raft_consensus.cc:3037] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:49.139091 18188 raft_consensus.cc:2454] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
I20260807 03:39:49.130290 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:49.142436 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:49.144690 18183 raft_consensus.cc:1226] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 2. Current term is 3. Ops: []
I20260807 03:39:49.145246 18180 raft_consensus.cc:2379] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 7: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:49.145746 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:49.152514 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:49.155995 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:49.157861 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:49.137061 22760 raft_consensus.cc:504] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:49.160600 18140 leader_election.cc:398] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Vote denied by peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163) with higher term. Message: Invalid argument: T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 7. Current term is 8.
I20260807 03:39:49.160879 18140 leader_election.cc:401] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260807 03:39:49.162015 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:49.162928 18180 raft_consensus.cc:2379] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 13: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:49.163964 18185 raft_consensus.cc:3032] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Stepping down as leader of term 6
I20260807 03:39:49.165910 18185 raft_consensus.cc:729] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:49.169981 18183 raft_consensus.cc:1226] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:49.159523 22658 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 13 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:49.171864 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:49.173527 18180 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
mode: GRACEFUL
new_leader_uuid: "66a36e1ac8e945e893668e282d394c49"
from {username='slave'} at 127.0.0.1:34758
I20260807 03:39:49.176870 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:49.178267 18069 raft_consensus.cc:1261] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260807 03:39:49.175423 18180 raft_consensus.cc:595] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 LEADER]: Received request to transfer leadership to TS 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:49.181473 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ca0223f47b46fd955704eba5259370" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:49.182066 22257 raft_consensus.cc:3037] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Advancing to term 8
I20260807 03:39:49.180028 22476 leader_election.cc:288] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:49.180759 18185 consensus_queue.cc:260] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:49.190706 18141 leader_election.cc:302] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:49.197299 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:49.183501 22748 leader_election.cc:288] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:49.198630 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:49.199591 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
W20260807 03:39:49.200918 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 63635.7us
I20260807 03:39:49.199043 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:49.204866 18185 raft_consensus.cc:3037] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:49.205446 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:49.204829 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:49.207419 18072 raft_consensus.cc:3037] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Advancing to term 11
I20260807 03:39:49.209851 18183 raft_consensus.cc:3032] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 03:39:49.210248 18183 raft_consensus.cc:729] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:49.200225 22476 raft_consensus.cc:2781] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 03:39:49.215688 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:49.219146 18190 raft_consensus.cc:1261] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260807 03:39:49.196177 22530 raft_consensus.cc:504] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:49.202708 22814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:46882 (request call id 174) took 1295 ms. Trace:
I20260807 03:39:49.222352 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:49.207388 22748 raft_consensus.cc:482] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:49.224418 18081 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a"
dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
mode: GRACEFUL
new_leader_uuid: "66a36e1ac8e945e893668e282d394c49"
from {username='slave'} at 127.0.0.1:55742
I20260807 03:39:49.226045 18081 raft_consensus.cc:595] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Received request to transfer leadership to TS 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:49.226742 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:49.213683 22476 raft_consensus.cc:482] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:49.218241 22760 leader_election.cc:288] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 11 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:49.233330 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:49.236307 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:49.237035 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:49.228609 18183 consensus_queue.cc:260] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:49.243232 18183 raft_consensus.cc:3037] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:49.232230 22476 raft_consensus.cc:3037] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:49.246408 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:49.247609 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260807 03:39:49.253403 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:49.251449 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:39:49.249869 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:49.254112 18312 raft_consensus.cc:2454] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 7.
I20260807 03:39:49.256708 18249 leader_election.cc:302] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:49.237695 22748 raft_consensus.cc:504] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:49.240182 22774 consensus_queue.cc:1035] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.013s
I20260807 03:39:49.252315 22657 consensus_queue.cc:1046] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20260807 03:39:49.263197 22817 raft_consensus.cc:2781] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 03:39:49.263588 22817 raft_consensus.cc:482] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:49.263945 22817 raft_consensus.cc:3037] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:49.266535 18072 raft_consensus.cc:2454] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 11.
I20260807 03:39:49.266657 18185 raft_consensus.cc:2454] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 7.
I20260807 03:39:49.270999 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:49.270684 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:49.268234 19239 leader_election.cc:302] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:49.268481 18081 raft_consensus.cc:2454] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 8.
I20260807 03:39:49.276837 18298 raft_consensus.cc:2454] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 8.
I20260807 03:39:49.272693 18071 raft_consensus.cc:2454] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 7.
I20260807 03:39:49.279560 18249 leader_election.cc:302] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:49.268898 22441 raft_consensus.cc:3032] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Stepping down as leader of term 6
I20260807 03:39:49.287799 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:49.293987 18298 raft_consensus.cc:1226] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 5. Current term is 6. Ops: [5.3-5.3]
W20260807 03:39:49.292948 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 64456.7us
I20260807 03:39:49.285110 22441 raft_consensus.cc:729] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:49.300426 18249 leader_election.cc:302] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
W20260807 03:39:49.293105 18027 connection.cc:232] RPC call timeout handler was delayed by -1.2299011701252311s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1698s for a response
W20260807 03:39:49.306380 18027 connection.cc:232] RPC call timeout handler was delayed by -1.084345315117389s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17210000000000003s for a response
I20260807 03:39:49.312148 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:49.304679 22532 raft_consensus.cc:2781] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:39:49.302153 22530 leader_election.cc:288] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:49.322690 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 100336us
I20260807 03:39:49.322052 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:49.223363 22814 rpcz_store.cc:276] 0807 03:39:47.921888 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:47.941970 (+ 20082us) service_pool.cc:227] Handling call
0807 03:39:48.631755 (+689785us) ts_tablet_manager.cc:928] T 6b66966247c147308a4b8488557e0bdf P 5c9b5062bdfa46af9396f1a85025a331: Initiating tablet copy from peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163)
0807 03:39:49.059496 (+427741us) tablet_copy_client.cc:361] Tablet copy session begun
0807 03:39:49.075624 (+ 16128us) tablet_metadata.cc:641] Metadata flushed
0807 03:39:49.075891 (+ 267us) tablet_copy_client.cc:490] Wrote new tablet metadata
0807 03:39:49.137430 (+ 61539us) tablet_copy_client.cc:411] Wrote new consensus metadata
0807 03:39:49.164573 (+ 27143us) tablet_replica.cc:205] Creating consensus instance
0807 03:39:49.181801 (+ 17228us) ts_tablet_manager.cc:983] Replica registered.
0807 03:39:49.187249 (+ 5448us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":505,"tablet-copy.queue_time_us":665574,"thread_start_us":1011201,"threads_started":2}
I20260807 03:39:49.324311 18141 leader_election.cc:302] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:49.325222 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:49.326366 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:49.327039 18080 raft_consensus.cc:3037] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:49.333158 18073 raft_consensus.cc:2379] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 7: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:49.328627 18071 raft_consensus.cc:3037] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 7
I20260807 03:39:49.315851 22520 raft_consensus.cc:2781] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 03:39:49.338137 22750 raft_consensus.cc:2781] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [term 11 FOLLOWER]: Leader election won for term 11
I20260807 03:39:49.346055 22441 consensus_queue.cc:260] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:49.347653 22441 raft_consensus.cc:3037] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:49.352007 18082 raft_consensus.cc:3037] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:49.352870 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:49.353785 18079 raft_consensus.cc:3037] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 11 FOLLOWER]: Advancing to term 13
I20260807 03:39:49.358053 22257 raft_consensus.cc:2726] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163) with higher term. Message: Invalid argument: T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 7. Current term is 8.
I20260807 03:39:49.317963 22532 raft_consensus.cc:482] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:49.359854 22532 raft_consensus.cc:3037] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:49.365459 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:39:49.371690 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:49.362964 22750 raft_consensus.cc:686] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [term 11 LEADER]: Becoming Leader. State: Replica: 5c9b5062bdfa46af9396f1a85025a331, State: Running, Role: LEADER
I20260807 03:39:49.374423 22750 consensus_queue.cc:237] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:49.381014 22288 raft_consensus.cc:979] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d: : Instructing follower 3c4bceb836db4536a0f255fdf86abc02 to start an election
I20260807 03:39:49.381503 22288 raft_consensus.cc:1067] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Signalling peer 3c4bceb836db4536a0f255fdf86abc02 to start an election
I20260807 03:39:49.383764 22132 consensus_queue.cc:1035] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:39:49.385735 18027 leader_election.cc:302] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:49.361759 22814 tablet_copy_client.cc:791] T 6b66966247c147308a4b8488557e0bdf P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Starting download of 0 data blocks...
I20260807 03:39:49.386955 22814 tablet_copy_client.cc:655] T 6b66966247c147308a4b8488557e0bdf P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Starting download of 1 WAL segments...
I20260807 03:39:49.392940 22814 tablet_copy_client.cc:531] T 6b66966247c147308a4b8488557e0bdf P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 03:39:49.388223 18027 leader_election.cc:398] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 election: Vote denied by peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057) with higher term. Message: Invalid argument: T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 6. Current term is 7.
I20260807 03:39:49.394682 18027 leader_election.cc:401] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260807 03:39:49.396178 18249 leader_election.cc:302] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
W20260807 03:39:49.409072 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 97671.1us
I20260807 03:39:49.343963 22520 raft_consensus.cc:482] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:49.417517 18072 raft_consensus.cc:3037] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Advancing to term 10
I20260807 03:39:49.420113 22288 raft_consensus.cc:2726] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057) with higher term. Message: Invalid argument: T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 6. Current term is 7.
I20260807 03:39:49.430863 18141 leader_election.cc:302] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:49.431180 17912 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35098 (request call id 82) took 4329 ms. Trace:
I20260807 03:39:49.431576 17912 rpcz_store.cc:276] 0807 03:39:45.135742 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:45.139636 (+ 3894us) service_pool.cc:227] Handling call
0807 03:39:49.430926 (+4291290us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1}
I20260807 03:39:49.432035 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:49.433023 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:49.433350 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:49.434649 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:49.435670 22774 raft_consensus.cc:2726] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:49.439137 19240 leader_election.cc:302] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:49.439533 18141 leader_election.cc:302] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:39:49.440069 22760 raft_consensus.cc:2781] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:49.440469 22760 raft_consensus.cc:482] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:49.440886 22760 raft_consensus.cc:3037] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:49.441964 17915 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39494 (request call id 197) took 4419 ms. Trace:
I20260807 03:39:49.450115 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:49.450480 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 104931us
I20260807 03:39:49.451349 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:49.451778 18249 leader_election.cc:302] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:49.452064 18309 raft_consensus.cc:2454] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 6.
I20260807 03:39:49.452468 17920 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39510 (request call id 202) took 4230 ms. Trace:
I20260807 03:39:49.453186 22685 raft_consensus.cc:2781] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:49.453860 17920 rpcz_store.cc:276] 0807 03:39:45.222155 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:45.222371 (+ 216us) service_pool.cc:227] Handling call
0807 03:39:49.452183 (+4229812us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":5}
I20260807 03:39:49.454275 22685 raft_consensus.cc:482] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:49.456002 22685 raft_consensus.cc:3037] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:49.460263 22827 raft_consensus.cc:2781] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:49.460842 22827 raft_consensus.cc:482] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:49.442399 17915 rpcz_store.cc:276] 0807 03:39:45.008821 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:45.009179 (+ 358us) service_pool.cc:227] Handling call
0807 03:39:49.427756 (+4418577us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 03:39:48.898463 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 03:39:48.899079 (+ 616us) write_op.cc:432] Acquiring schema lock in shared mode
0807 03:39:48.899145 (+ 66us) write_op.cc:435] Acquired schema lock
0807 03:39:48.899162 (+ 17us) tablet.cc:660] Decoding operations
0807 03:39:48.899523 (+ 361us) write_op.cc:620] Acquiring the partition lock for write op
0807 03:39:48.899651 (+ 128us) write_op.cc:641] Partition lock acquired for write op
0807 03:39:48.899693 (+ 42us) tablet.cc:683] Acquiring locks for 1 operations
0807 03:39:48.899939 (+ 246us) tablet.cc:699] Row locks acquired
0807 03:39:48.899982 (+ 43us) write_op.cc:260] PREPARE: finished
0807 03:39:48.909335 (+ 9353us) write_op.cc:270] Start()
0807 03:39:48.960118 (+ 50783us) write_op.cc:276] Timestamp: P: 1786073988904707 usec, L: 0
0807 03:39:48.966421 (+ 6303us) op_driver.cc:331] REPLICATION: starting
0807 03:39:49.008598 (+ 42177us) log.cc:842] Serialized 544 byte log entry
0807 03:39:49.220652 (+212054us) op_driver.cc:444] REPLICATION: finished
0807 03:39:49.419709 (+199057us) write_op.cc:301] APPLY: starting
0807 03:39:49.419841 (+ 132us) tablet.cc:1365] starting BulkCheckPresence
0807 03:39:49.419966 (+ 125us) tablet.cc:1368] finished BulkCheckPresence
0807 03:39:49.420124 (+ 158us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 03:39:49.420536 (+ 412us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 03:39:49.420874 (+ 338us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0807 03:39:49.420925 (+ 51us) write_op.cc:312] APPLY: finished
0807 03:39:49.421318 (+ 393us) log.cc:842] Serialized 23 byte log entry
0807 03:39:49.421473 (+ 155us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:39:49.421554 (+ 81us) write_op.cc:454] Released schema lock
0807 03:39:49.421617 (+ 63us) write_op.cc:341] FINISH: Updating metrics
0807 03:39:49.421976 (+ 359us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:39:49.422005 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":8,"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":144824,"apply.run_cpu_time_us":2645,"apply.run_wall_time_us":3321,"num_ops":1,"prepare.queue_time_us":53773,"prepare.run_cpu_time_us":3557,"prepare.run_wall_time_us":126088,"replication_time_us":251192,"thread_start_us":106186,"threads_started":2}]]}
I20260807 03:39:49.452759 18249 leader_election.cc:302] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:39:49.470777 22426 raft_consensus.cc:2726] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 03:39:49.472961 18248 leader_election.cc:398] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Vote denied by peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163) with higher term. Message: Invalid argument: T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 7. Current term is 8.
I20260807 03:39:49.472324 18249 leader_election.cc:302] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:49.474306 18079 raft_consensus.cc:2454] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 13.
I20260807 03:39:49.419270 22520 raft_consensus.cc:3037] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:49.440464 22532 raft_consensus.cc:504] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:49.477923 22774 raft_consensus.cc:2781] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:39:49.478353 22774 raft_consensus.cc:482] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 5c9b5062bdfa46af9396f1a85025a331)
I20260807 03:39:49.478701 22774 raft_consensus.cc:3037] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:49.461254 22827 raft_consensus.cc:3037] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:49.461795 22748 leader_election.cc:288] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
W20260807 03:39:49.462138 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 67994.9us
I20260807 03:39:49.439266 22817 raft_consensus.cc:504] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:49.487857 18082 raft_consensus.cc:2454] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 7.
I20260807 03:39:49.524170 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:49.524525 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:49.525087 18180 raft_consensus.cc:2379] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 7: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:49.525380 18190 raft_consensus.cc:2354] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 12. Current term is 13.
W20260807 03:39:49.520684 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 48506.5us
W20260807 03:39:49.362324 22658 consensus_queue.cc:1159] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "66a36e1ac8e945e893668e282d394c49" responder_term: 6 status { last_received { term: 6 index: 4 } last_committed_idx: 4 last_received_current_leader { term: 6 index: 4 } }
I20260807 03:39:49.537724 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:49.538599 18185 raft_consensus.cc:2379] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 7: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:49.533026 22748 raft_consensus.cc:2781] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:49.540705 22748 raft_consensus.cc:482] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:49.541183 22748 raft_consensus.cc:3037] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:49.543493 22817 leader_election.cc:288] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:49.545806 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:49.547401 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:49.548439 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:49.548647 18184 raft_consensus.cc:1261] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
W20260807 03:39:49.549305 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 161213us
I20260807 03:39:49.549834 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:49.561374 22476 raft_consensus.cc:504] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:49.562206 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 122861us
I20260807 03:39:49.550541 18248 leader_election.cc:302] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:49.564476 22532 leader_election.cc:288] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:49.562247 22762 raft_consensus.cc:2781] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 03:39:49.579169 22762 raft_consensus.cc:482] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:49.579586 22762 raft_consensus.cc:3037] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:49.580651 18027 leader_election.cc:302] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:49.583925 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:49.584087 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:49.584558 18311 raft_consensus.cc:3037] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:49.565788 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:49.587301 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:49.588042 18309 raft_consensus.cc:2379] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 7: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:49.588280 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:49.588459 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:49.588850 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:49.589408 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:49.589582 18312 raft_consensus.cc:2379] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 9: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:49.589613 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:49.590472 18190 raft_consensus.cc:2379] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 8: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:49.591012 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:49.591627 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:49.592308 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:49.593055 18189 raft_consensus.cc:2454] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 6.
I20260807 03:39:49.571573 18183 raft_consensus.cc:2454] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:49.575261 18072 raft_consensus.cc:2454] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 10.
I20260807 03:39:49.576787 18141 leader_election.cc:302] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
W20260807 03:39:49.625389 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 48696.6us
I20260807 03:39:49.626483 18141 leader_election.cc:302] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:49.615960 22762 raft_consensus.cc:504] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:39:49.627350 18141 leader_election.cc:302] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:49.592392 18184 raft_consensus.cc:2454] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 7.
I20260807 03:39:49.599582 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:49.597343 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:49.593837 18248 leader_election.cc:302] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:49.593701 22817 raft_consensus.cc:2726] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:49.600569 18309 raft_consensus.cc:1226] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:49.594425 22427 consensus_queue.cc:1035] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 03:39:49.604679 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:49.604825 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:49.632323 22441 raft_consensus.cc:2726] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 03:39:49.635250 22781 raft_consensus.cc:2726] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:49.632107 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:49.633028 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ca0223f47b46fd955704eba5259370" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:49.637501 18185 raft_consensus.cc:2379] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 3: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:49.637554 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:49.607630 18071 raft_consensus.cc:2454] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 7.
I20260807 03:39:49.609032 18311 raft_consensus.cc:2454] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 8.
I20260807 03:39:49.638487 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:49.638327 22748 raft_consensus.cc:504] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:49.609218 22827 raft_consensus.cc:504] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:49.640024 18191 raft_consensus.cc:1226] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 4. Current term is 5. Ops: []
I20260807 03:39:49.640832 22748 leader_election.cc:288] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:49.641780 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:49.634567 18248 leader_election.cc:302] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:49.642841 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:49.643818 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:49.639731 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:49.644961 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:49.645593 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:49.647231 18191 raft_consensus.cc:1226] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 7. Current term is 8. Ops: []
I20260807 03:39:49.648643 18082 raft_consensus.cc:1226] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 11 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 10. Current term is 11. Ops: []
I20260807 03:39:49.649276 22827 leader_election.cc:288] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:49.611418 22814 tablet_bootstrap.cc:492] T 6b66966247c147308a4b8488557e0bdf P 5c9b5062bdfa46af9396f1a85025a331: Bootstrap starting.
I20260807 03:39:49.598243 22476 leader_election.cc:288] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:49.593508 22788 raft_consensus.cc:504] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:49.619352 19292 raft_consensus.cc:1226] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331 [term 7 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 6. Current term is 7. Ops: []
I20260807 03:39:49.620416 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:49.622355 22532 consensus_queue.cc:1046] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:39:49.651302 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:49.651525 18193 raft_consensus.cc:2454] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 8.
I20260807 03:39:49.651966 22532 raft_consensus.cc:2726] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260807 03:39:49.652376 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:49.652580 22532 raft_consensus.cc:3032] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 03:39:49.652983 22532 raft_consensus.cc:729] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:49.653541 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:49.653811 22532 consensus_queue.cc:260] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:49.654422 22788 leader_election.cc:288] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:49.654747 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:49.655068 22532 raft_consensus.cc:3037] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:49.623023 22426 consensus_queue.cc:1046] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 03:39:49.623616 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:49.647943 18191 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1f5db05c008b404288067417c3f100d2"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
from {username='slave'} at 127.0.0.1:44662
I20260807 03:39:49.598598 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:49.656487 18191 raft_consensus.cc:482] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:39:49.657028 18191 raft_consensus.cc:3037] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:49.657241 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:49.631958 22658 raft_consensus.cc:2781] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 03:39:49.658980 22658 raft_consensus.cc:482] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:49.659350 22658 raft_consensus.cc:3037] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:49.622928 22685 raft_consensus.cc:504] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:49.638345 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:49.660691 18185 raft_consensus.cc:2379] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 4: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:49.638516 22132 raft_consensus.cc:2741] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260807 03:39:49.661499 22685 leader_election.cc:288] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:49.622848 18249 leader_election.cc:302] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:39:49.648517 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:49.653693 18180 raft_consensus.cc:1226] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:49.662881 22427 raft_consensus.cc:2726] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 03:39:49.663487 22427 consensus_queue.cc:1035] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 03:39:49.665328 18069 raft_consensus.cc:1226] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260807 03:39:49.652681 18193 raft_consensus.cc:1226] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260807 03:39:49.628077 22257 raft_consensus.cc:2781] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:49.656545 22781 raft_consensus.cc:3032] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:39:49.666317 18248 leader_election.cc:302] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:49.606776 22760 raft_consensus.cc:504] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:49.673038 22817 consensus_queue.cc:1046] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260807 03:39:49.673617 18249 leader_election.cc:302] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:49.674031 22426 raft_consensus.cc:3032] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 6 LEADER]: Stepping down as leader of term 6
I20260807 03:39:49.674319 22817 raft_consensus.cc:2781] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 03:39:49.674461 22426 raft_consensus.cc:729] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:49.673957 17918 catalog_manager.cc:5495] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 4 to 5, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 5 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:49.675335 22426 consensus_queue.cc:260] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:49.677265 18080 raft_consensus.cc:2454] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:49.677896 22776 raft_consensus.cc:2726] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:49.679026 22426 raft_consensus.cc:3037] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:49.679764 18248 leader_election.cc:302] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
W20260807 03:39:49.680084 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 52809.7us
I20260807 03:39:49.680708 18249 leader_election.cc:302] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:49.668201 22781 raft_consensus.cc:729] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:49.636508 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:49.668578 18081 raft_consensus.cc:1226] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260807 03:39:49.669026 22257 raft_consensus.cc:482] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:49.682267 22257 raft_consensus.cc:3037] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:49.682443 22760 leader_election.cc:288] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:49.570277 22774 raft_consensus.cc:504] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:49.685356 18185 raft_consensus.cc:1226] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 6. Current term is 7. Ops: []
I20260807 03:39:49.686259 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:49.687259 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:49.687937 18185 raft_consensus.cc:2379] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 5: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:49.693650 22257 raft_consensus.cc:504] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:49.694793 18190 raft_consensus.cc:1226] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 4. Current term is 5. Ops: []
I20260807 03:39:49.670677 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:49.670791 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:49.671504 18187 raft_consensus.cc:2454] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 8.
I20260807 03:39:49.672178 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:49.672417 22427 raft_consensus.cc:2781] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 03:39:49.699728 18073 raft_consensus.cc:2379] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 7: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:49.681915 22781 consensus_queue.cc:260] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:49.669443 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:49.669998 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:49.689352 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:49.689843 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:49.700557 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:49.690335 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:49.701710 18180 raft_consensus.cc:2379] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 5: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:49.690637 19292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331"
I20260807 03:39:49.691200 22774 leader_election.cc:288] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Requested vote from peers 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:49.693449 18189 raft_consensus.cc:1226] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 7. Current term is 8. Ops: []
I20260807 03:39:49.697192 18082 raft_consensus.cc:2379] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 13: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:49.680801 18141 leader_election.cc:302] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:49.699656 18183 raft_consensus.cc:1226] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Rejecting Update request from peer 5c9b5062bdfa46af9396f1a85025a331 for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:49.702709 18079 raft_consensus.cc:2379] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 8: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:49.701670 18184 raft_consensus.cc:2454] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 7.
W20260807 03:39:49.667603 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 48995.7us
I20260807 03:39:49.704757 22748 raft_consensus.cc:2781] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 03:39:49.699890 22427 raft_consensus.cc:482] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:49.715763 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:49.728729 22427 raft_consensus.cc:3037] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:49.717132 18070 raft_consensus.cc:1226] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 8. Current term is 9. Ops: []
I20260807 03:39:49.718102 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:49.719015 22426 raft_consensus.cc:2781] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 03:39:49.719100 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ca0223f47b46fd955704eba5259370" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:49.720376 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:49.724579 22832 consensus_queue.cc:1046] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260807 03:39:49.735893 18141 leader_election.cc:302] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:49.738930 22426 raft_consensus.cc:686] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:49.740147 22426 consensus_queue.cc:237] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:49.742704 18069 raft_consensus.cc:1226] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 8. Current term is 9. Ops: []
I20260807 03:39:49.719893 22781 raft_consensus.cc:3037] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:49.739567 22257 leader_election.cc:288] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:49.718838 22762 leader_election.cc:288] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:49.751058 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:49.752986 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:49.753567 18183 raft_consensus.cc:3037] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:49.762964 22842 consensus_queue.cc:1046] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 03:39:49.767373 18071 raft_consensus.cc:1226] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 4. Current term is 5. Ops: []
I20260807 03:39:49.738355 22748 raft_consensus.cc:482] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:49.756078 22257 raft_consensus.cc:2781] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 03:39:49.752933 22832 raft_consensus.cc:3032] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 03:39:49.776279 22832 raft_consensus.cc:729] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 5c9b5062bdfa46af9396f1a85025a331, State: Running, Role: LEADER
I20260807 03:39:49.777103 22832 consensus_queue.cc:260] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:49.778260 22832 raft_consensus.cc:3037] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:49.779927 18183 raft_consensus.cc:2454] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 9.
I20260807 03:39:49.774564 22476 consensus_queue.cc:1046] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20260807 03:39:49.784086 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:49.786685 22257 raft_consensus.cc:686] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:49.787649 22257 consensus_queue.cc:237] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:49.786216 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
W20260807 03:39:49.791837 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 112141us
I20260807 03:39:49.770313 18027 leader_election.cc:302] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:49.763188 18191 raft_consensus.cc:504] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:49.750736 22817 raft_consensus.cc:686] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:49.794274 22817 consensus_queue.cc:237] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:49.796365 22788 raft_consensus.cc:2741] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260807 03:39:49.795835 18248 leader_election.cc:302] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:49.805200 19295 raft_consensus.cc:1226] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331 [term 4 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:49.805483 22520 raft_consensus.cc:504] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:49.806648 22658 raft_consensus.cc:504] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:49.807461 22520 leader_election.cc:288] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:49.808393 22658 leader_election.cc:288] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:49.809190 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:49.812398 22132 raft_consensus.cc:2781] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260807 03:39:49.812239 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 51257.5us
I20260807 03:39:49.816502 22132 raft_consensus.cc:482] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:49.811976 18191 leader_election.cc:288] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:49.816929 22132 raft_consensus.cc:3037] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:49.818866 19292 raft_consensus.cc:1226] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [term 11 LEADER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 10. Current term is 11. Ops: []
W20260807 03:39:49.820277 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RunLeaderElection -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 65883.7us
I20260807 03:39:49.792766 22748 raft_consensus.cc:3037] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:49.822932 18140 leader_election.cc:302] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:49.831830 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:49.833863 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:49.834008 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:49.835110 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:49.836192 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:49.836477 18027 leader_election.cc:302] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:49.840318 18295 raft_consensus.cc:2379] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 7: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:49.842136 18027 leader_election.cc:302] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:39:49.842585 22288 raft_consensus.cc:2781] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:49.842995 22288 raft_consensus.cc:482] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:49.843365 22288 raft_consensus.cc:3037] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:49.844259 22776 raft_consensus.cc:2781] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 03:39:49.844628 22776 raft_consensus.cc:482] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5c9b5062bdfa46af9396f1a85025a331)
I20260807 03:39:49.844969 22776 raft_consensus.cc:3037] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:49.851711 22774 raft_consensus.cc:3032] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Stepping down as leader of term 6
I20260807 03:39:49.852110 22774 raft_consensus.cc:729] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:49.823805 22762 raft_consensus.cc:2726] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260807 03:39:49.862044 22132 raft_consensus.cc:504] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:49.821776 22739 consensus_queue.cc:1046] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.019s
W20260807 03:39:49.869632 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 105602us
I20260807 03:39:49.866578 17915 catalog_manager.cc:5495] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 reported cstate change: term changed from 9 to 11, leader changed from d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) to 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132). New cstate: current_term: 11 leader_uuid: "5c9b5062bdfa46af9396f1a85025a331" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 03:39:49.869737 22132 leader_election.cc:288] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:49.833148 22476 raft_consensus.cc:2726] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260807 03:39:49.877391 18140 leader_election.cc:302] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:49.827905 22658 consensus_queue.cc:1046] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.043s
I20260807 03:39:49.876283 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:49.876483 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:49.876667 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:49.875293 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:49.881350 18298 raft_consensus.cc:2379] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 13: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:49.830662 22788 consensus_queue.cc:1046] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.041s
I20260807 03:39:49.885811 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ca0223f47b46fd955704eba5259370" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:49.890489 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:49.891388 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:49.892968 18305 raft_consensus.cc:1226] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260807 03:39:49.890446 18249 leader_election.cc:302] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:49.897975 18140 leader_election.cc:302] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:49.900475 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:49.901069 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:49.887187 22827 raft_consensus.cc:2726] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 03:39:49.905174 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:49.905648 18297 raft_consensus.cc:3037] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:49.906550 18300 raft_consensus.cc:2379] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 6: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:49.904055 22685 raft_consensus.cc:2781] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader election won for term 9
I20260807 03:39:49.910426 18295 raft_consensus.cc:1226] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260807 03:39:49.919610 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:49.920378 18191 raft_consensus.cc:2379] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 8: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:49.924278 18310 raft_consensus.cc:2454] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
I20260807 03:39:49.894536 22762 consensus_queue.cc:1046] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20260807 03:39:49.897568 22441 consensus_queue.cc:1046] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.020s
I20260807 03:39:49.909652 22658 raft_consensus.cc:2726] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:49.937731 18297 raft_consensus.cc:2454] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 6.
I20260807 03:39:49.939684 22288 raft_consensus.cc:504] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:49.942339 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 98424.8us
I20260807 03:39:49.918457 22685 raft_consensus.cc:686] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 9 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:49.946266 22288 leader_election.cc:288] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:49.932114 22441 consensus_queue.cc:1225] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer 5c9b5062bdfa46af9396f1a85025a331 log is divergent from this leader: its last log entry 11.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260807 03:39:49.939725 22762 raft_consensus.cc:3032] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Stepping down as leader of term 7
I20260807 03:39:49.947501 22762 raft_consensus.cc:729] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:49.948390 22762 consensus_queue.cc:260] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:49.949556 22762 raft_consensus.cc:3037] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:49.909646 22774 consensus_queue.cc:260] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:49.957726 22788 consensus_queue.cc:1046] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 03:39:49.962818 22427 raft_consensus.cc:504] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:49.963981 18311 raft_consensus.cc:1226] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 8. Current term is 9. Ops: []
I20260807 03:39:49.970491 18075 raft_consensus.cc:1261] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 11 FOLLOWER]: Refusing update from remote peer 5c9b5062bdfa46af9396f1a85025a331: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260807 03:39:49.970724 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:49.975072 18293 raft_consensus.cc:2454] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
I20260807 03:39:49.966867 22857 raft_consensus.cc:3032] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:39:49.978453 22857 raft_consensus.cc:729] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:49.964432 22520 raft_consensus.cc:3032] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 6 LEADER]: Stepping down as leader of term 6
I20260807 03:39:49.979527 22520 raft_consensus.cc:729] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:49.979214 22857 consensus_queue.cc:260] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:49.980042 18185 raft_consensus.cc:3032] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 10 LEADER]: Stepping down as leader of term 10
I20260807 03:39:49.980424 18185 raft_consensus.cc:729] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:49.980674 22857 raft_consensus.cc:3037] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:49.980561 22520 consensus_queue.cc:260] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:49.981299 18185 consensus_queue.cc:260] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:49.982501 22520 raft_consensus.cc:3037] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:49.982522 18185 raft_consensus.cc:3037] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Advancing to term 11
I20260807 03:39:49.965027 22530 raft_consensus.cc:3032] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 LEADER]: Stepping down as leader of term 8
I20260807 03:39:49.991102 22530 raft_consensus.cc:729] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:49.991815 22750 consensus_queue.cc:1035] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260807 03:39:49.992249 22530 consensus_queue.cc:260] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:49.993861 22530 raft_consensus.cc:3037] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:49.964061 22776 raft_consensus.cc:504] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:49.997267 22441 consensus_queue.cc:1046] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 03:39:49.998728 19292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331"
W20260807 03:39:49.999163 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 54678.7us
I20260807 03:39:50.002331 22827 raft_consensus.cc:3032] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 LEADER]: Stepping down as leader of term 7
I20260807 03:39:50.002794 22827 raft_consensus.cc:729] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
W20260807 03:39:50.017947 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 84221.2us
W20260807 03:39:50.021746 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 219579us
I20260807 03:39:49.971572 22427 leader_election.cc:288] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:50.034888 18248 leader_election.cc:302] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:50.030350 18141 leader_election.cc:302] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:50.039019 18248 leader_election.cc:302] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:50.040920 18248 leader_election.cc:302] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:50.046231 18141 leader_election.cc:302] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:50.046451 22658 raft_consensus.cc:2726] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 03:39:50.046754 18248 leader_election.cc:302] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:50.047066 18141 leader_election.cc:302] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:50.047808 22658 raft_consensus.cc:2781] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:39:50.047955 22476 raft_consensus.cc:2781] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 03:39:50.048308 22838 raft_consensus.cc:2781] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 13 FOLLOWER]: Leader election won for term 13
I20260807 03:39:50.048993 22427 raft_consensus.cc:2726] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260807 03:39:50.049132 18248 leader_election.cc:302] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:50.054790 22871 raft_consensus.cc:2781] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Leader election won for term 10
I20260807 03:39:50.055550 22427 raft_consensus.cc:2781] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 03:39:49.983510 22748 raft_consensus.cc:504] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:50.001199 22776 leader_election.cc:288] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Requested vote from peers 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:50.055404 22871 raft_consensus.cc:686] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 10 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
W20260807 03:39:50.103391 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 58057.9us
I20260807 03:39:50.042956 22781 raft_consensus.cc:2726] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:50.113675 18309 raft_consensus.cc:1226] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 7. Current term is 8. Ops: []
I20260807 03:39:50.108546 22838 raft_consensus.cc:686] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [term 13 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:50.120131 18141 leader_election.cc:302] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
W20260807 03:39:50.127064 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 78194.8us
I20260807 03:39:50.003767 22827 consensus_queue.cc:260] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:50.113122 22427 raft_consensus.cc:686] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:50.114831 22476 raft_consensus.cc:686] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
W20260807 03:39:50.139307 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 91217.9us
I20260807 03:39:50.140034 18140 leader_election.cc:302] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:50.059552 22685 consensus_queue.cc:237] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:39:50.141075 22658 raft_consensus.cc:686] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:50.142140 22658 consensus_queue.cc:237] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:50.149430 19240 leader_election.cc:302] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:50.149942 18027 leader_election.cc:302] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:50.120730 18185 raft_consensus.cc:1261] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 11 FOLLOWER]: Refusing update from remote peer 5c9b5062bdfa46af9396f1a85025a331: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 9. (term mismatch)
I20260807 03:39:50.150725 18185 pending_rounds.cc:76] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02: Aborting all ops after (but not including) 8
I20260807 03:39:50.150982 18185 pending_rounds.cc:97] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02: Aborting uncommitted NO_OP operation due to leader change: 10.9
I20260807 03:39:50.153227 18185 raft_consensus.cc:2866] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260807 03:39:50.154872 22824 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 11 current term: 11 is higher.
I20260807 03:39:50.157850 22871 consensus_queue.cc:237] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:50.159781 22860 log.cc:924] Time spent T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260807 03:39:50.138492 22257 consensus_queue.cc:1046] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20260807 03:39:50.205477 18185 raft_consensus.cc:1261] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260807 03:39:50.208657 18027 leader_election.cc:302] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
W20260807 03:39:50.210505 19240 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 61170.1us
I20260807 03:39:50.178149 22739 raft_consensus.cc:2781] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 03:39:50.206327 22882 raft_consensus.cc:2726] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260807 03:39:50.216770 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 50177.5us
I20260807 03:39:50.217737 18140 leader_election.cc:302] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:50.219111 22750 raft_consensus.cc:2726] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260807 03:39:50.230841 18293 raft_consensus.cc:1261] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260807 03:39:50.216771 22739 raft_consensus.cc:482] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:50.233649 22739 raft_consensus.cc:3037] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:50.236191 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
W20260807 03:39:50.237699 22875 log.cc:924] Time spent T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.084s user 0.000s sys 0.000s
I20260807 03:39:50.241133 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:50.152352 22838 consensus_queue.cc:237] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:50.245677 22869 consensus_queue.cc:1046] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.080s
I20260807 03:39:50.246330 22863 raft_consensus.cc:2781] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 03:39:50.246766 22865 raft_consensus.cc:2781] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader election won for term 8
I20260807 03:39:50.223245 22748 leader_election.cc:288] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:50.248752 22788 raft_consensus.cc:2726] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260807 03:39:50.249167 22441 consensus_queue.cc:1035] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:39:50.249755 22132 consensus_queue.cc:1046] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:39:50.272042 18297 raft_consensus.cc:1226] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Rejecting Update request from peer 5c9b5062bdfa46af9396f1a85025a331 for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:50.222906 22426 raft_consensus.cc:482] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
W20260807 03:39:50.328076 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 48586.3us
I20260807 03:39:50.328656 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:50.329612 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:50.330554 18075 raft_consensus.cc:1226] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 7. Current term is 8. Ops: []
I20260807 03:39:50.331247 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:50.333137 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:50.334069 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:50.331530 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:50.212788 22427 consensus_queue.cc:237] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:50.326767 19240 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 114690us
I20260807 03:39:50.214699 22476 consensus_queue.cc:237] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:50.351747 19240 leader_election.cc:302] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:50.346756 18248 leader_election.cc:302] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:50.354588 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:50.355650 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
W20260807 03:39:50.360364 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 142704us
I20260807 03:39:50.361734 18140 leader_election.cc:302] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:50.366268 22889 raft_consensus.cc:2726] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 03:39:50.369329 18140 leader_election.cc:302] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:50.371031 18140 leader_election.cc:302] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:50.372303 18082 raft_consensus.cc:1226] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Rejecting Update request from peer 5c9b5062bdfa46af9396f1a85025a331 for earlier term 2. Current term is 3. Ops: []
I20260807 03:39:50.335577 22762 consensus_queue.cc:1035] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20260807 03:39:50.355881 22863 raft_consensus.cc:686] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:50.357926 22426 raft_consensus.cc:504] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:50.405040 18297 raft_consensus.cc:1226] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 10 LEADER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 9. Current term is 10. Ops: []
I20260807 03:39:50.399770 19240 leader_election.cc:302] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:50.404542 22132 raft_consensus.cc:3032] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Stepping down as leader of term 5
I20260807 03:39:50.410526 22132 raft_consensus.cc:729] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:50.383324 22824 raft_consensus.cc:2726] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260807 03:39:50.415266 22760 raft_consensus.cc:2781] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:39:50.415867 22760 raft_consensus.cc:482] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:50.416328 22760 raft_consensus.cc:3037] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:50.415099 22862 raft_consensus.cc:2932] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 8 LEADER]: Committing config change with OpId 8.7: config changed from index -1 to 7, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
W20260807 03:39:50.418766 22876 log.cc:924] Time spent T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.250s user 0.002s sys 0.000s
I20260807 03:39:50.415287 22885 consensus_queue.cc:1046] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
I20260807 03:39:50.415467 22881 raft_consensus.cc:2726] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260807 03:39:50.400282 22441 raft_consensus.cc:2726] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260807 03:39:50.424844 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 54059.7us
I20260807 03:39:50.407590 22891 raft_consensus.cc:2726] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 5c9b5062bdfa46af9396f1a85025a331 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260807 03:39:50.449795 22132 consensus_queue.cc:260] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:50.454254 19240 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 54604.6us
I20260807 03:39:50.455032 19240 leader_election.cc:302] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:50.455827 22832 raft_consensus.cc:2726] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 03:39:50.407552 22906 consensus_queue.cc:1046] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.056s
I20260807 03:39:50.460310 22132 raft_consensus.cc:3037] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:50.461539 18140 leader_election.cc:302] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:50.453212 22532 raft_consensus.cc:482] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:50.465499 22532 raft_consensus.cc:504] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:50.467317 22532 leader_election.cc:288] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:50.468437 22906 raft_consensus.cc:3032] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 03:39:50.468829 22906 raft_consensus.cc:729] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5c9b5062bdfa46af9396f1a85025a331, State: Running, Role: LEADER
I20260807 03:39:50.469669 22906 consensus_queue.cc:260] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
W20260807 03:39:50.471338 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 61781.4us
I20260807 03:39:50.475474 22906 raft_consensus.cc:3037] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:50.407616 22604 consensus_queue.cc:1035] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.213s
I20260807 03:39:50.475243 22739 raft_consensus.cc:504] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:50.480839 18140 leader_election.cc:302] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:50.501317 22739 leader_election.cc:288] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:50.455339 22865 raft_consensus.cc:686] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:50.505332 22865 consensus_queue.cc:237] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:50.505860 22426 leader_election.cc:288] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
W20260807 03:39:50.507469 22520 consensus_meta.cc:287] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.525s user 0.008s sys 0.002s
I20260807 03:39:50.508574 22774 raft_consensus.cc:3037] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:50.458931 22863 consensus_queue.cc:237] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:50.518885 22896 raft_consensus.cc:2726] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260807 03:39:50.556234 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:39:50.557104 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 65976.6us
I20260807 03:39:50.558409 18140 leader_election.cc:302] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
W20260807 03:39:50.563336 19240 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 105986us
W20260807 03:39:50.568497 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 317705us
I20260807 03:39:50.574571 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:50.578697 18185 raft_consensus.cc:2379] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 7: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:50.581171 18140 leader_election.cc:302] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:50.583719 18140 leader_election.cc:302] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:50.585227 18027 leader_election.cc:302] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:50.592540 22897 consensus_queue.cc:1046] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.111s
I20260807 03:39:50.593465 22853 raft_consensus.cc:482] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:50.593986 22853 raft_consensus.cc:504] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:50.594826 22918 raft_consensus.cc:2726] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 03:39:50.583530 22882 consensus_queue.cc:1046] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.012s
I20260807 03:39:50.596127 22902 consensus_queue.cc:1046] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.014s
I20260807 03:39:50.596555 22921 raft_consensus.cc:2726] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260807 03:39:50.600054 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:50.600675 18191 raft_consensus.cc:2454] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 9.
I20260807 03:39:50.588606 22885 raft_consensus.cc:2726] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 03:39:50.581525 22871 raft_consensus.cc:482] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:50.617579 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:50.584468 22863 raft_consensus.cc:2726] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 03:39:50.619751 18249 leader_election.cc:302] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:50.623665 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:50.627480 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:50.628166 18297 raft_consensus.cc:2379] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 8: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:50.626147 18298 raft_consensus.cc:1226] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 9 LEADER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 8. Current term is 9. Ops: []
I20260807 03:39:50.630672 18185 raft_consensus.cc:1261] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260807 03:39:50.632655 18300 raft_consensus.cc:1226] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 4. Current term is 5. Ops: []
I20260807 03:39:50.633267 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
W20260807 03:39:50.634598 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 287887us
I20260807 03:39:50.635882 22882 consensus_queue.cc:1225] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer 66a36e1ac8e945e893668e282d394c49 log is divergent from this leader: its last log entry 10.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260807 03:39:50.636232 22908 raft_consensus.cc:3032] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 8 LEADER]: Stepping down as leader of term 8
I20260807 03:39:50.636475 22882 raft_consensus.cc:3032] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 9 LEADER]: Stepping down as leader of term 9
I20260807 03:39:50.636612 22908 raft_consensus.cc:729] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:50.636830 22882 raft_consensus.cc:729] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:50.637466 22908 consensus_queue.cc:260] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:50.637609 22882 consensus_queue.cc:260] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:50.638936 22896 consensus_queue.cc:1159] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "66a36e1ac8e945e893668e282d394c49" responder_term: 8 status { last_received { term: 8 index: 6 } last_committed_idx: 6 last_received_current_leader { term: 8 index: 6 } }
I20260807 03:39:50.622375 22871 raft_consensus.cc:504] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:50.641327 18293 raft_consensus.cc:1226] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 4. Current term is 5. Ops: []
I20260807 03:39:50.641678 22871 leader_election.cc:288] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:50.638734 22882 raft_consensus.cc:3037] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Advancing to term 10
I20260807 03:39:50.645857 18297 raft_consensus.cc:1226] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 7. Current term is 8. Ops: []
I20260807 03:39:50.646632 18312 raft_consensus.cc:1226] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:50.647312 22530 raft_consensus.cc:3032] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 LEADER]: Stepping down as leader of term 8
I20260807 03:39:50.647676 22530 raft_consensus.cc:729] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:50.652086 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:50.652864 18189 raft_consensus.cc:2379] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 6: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:50.653309 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:50.652812 18248 leader_election.cc:302] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:50.656544 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:50.656689 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:50.657292 18191 raft_consensus.cc:2354] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 12. Current term is 13.
I20260807 03:39:50.657665 22827 raft_consensus.cc:3037] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:50.657889 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:50.659260 22427 consensus_queue.cc:1035] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260807 03:39:50.659525 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:50.659718 18081 raft_consensus.cc:1203] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 11 FOLLOWER]: Deduplicated request from leader. Original: 9.8->[10.9-10.9] Dedup: 10.9->[]
I20260807 03:39:50.660064 18071 raft_consensus.cc:3032] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Stepping down as leader of term 5
I20260807 03:39:50.660298 18081 raft_consensus.cc:1226] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 11 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260807 03:39:50.660455 18071 raft_consensus.cc:729] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:50.661230 18079 raft_consensus.cc:1226] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 4. Current term is 5. Ops: []
I20260807 03:39:50.661201 18071 consensus_queue.cc:260] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:50.662241 18071 raft_consensus.cc:3037] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:50.614652 22853 leader_election.cc:288] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:50.641275 22881 raft_consensus.cc:3032] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 LEADER]: Stepping down as leader of term 7
I20260807 03:39:50.668184 22881 raft_consensus.cc:729] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:50.608399 22760 raft_consensus.cc:504] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:50.668648 22530 consensus_queue.cc:260] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:50.668970 22881 consensus_queue.cc:260] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:50.670023 22760 leader_election.cc:288] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 7 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:50.670200 22530 raft_consensus.cc:3037] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:50.663352 22908 raft_consensus.cc:3037] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:50.672286 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:50.663599 18248 leader_election.cc:302] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:50.673951 19292 raft_consensus.cc:1226] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331 [term 7 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 6. Current term is 7. Ops: []
I20260807 03:39:50.664142 22916 raft_consensus.cc:2781] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260807 03:39:50.664520 22929 raft_consensus.cc:2781] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader election won for term 8
I20260807 03:39:50.666028 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:50.684286 22929 raft_consensus.cc:686] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:50.685272 22929 consensus_queue.cc:237] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:50.688829 18082 raft_consensus.cc:1226] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 4. Current term is 5. Ops: []
I20260807 03:39:50.676398 22871 raft_consensus.cc:2726] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260807 03:39:50.676287 22916 raft_consensus.cc:482] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:50.693012 22916 raft_consensus.cc:3037] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Advancing to term 10
I20260807 03:39:50.693725 22853 raft_consensus.cc:2726] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260807 03:39:50.697077 22814 tablet_bootstrap.cc:492] T 6b66966247c147308a4b8488557e0bdf P 5c9b5062bdfa46af9396f1a85025a331: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260807 03:39:50.698019 22814 tablet_bootstrap.cc:492] T 6b66966247c147308a4b8488557e0bdf P 5c9b5062bdfa46af9396f1a85025a331: Bootstrap complete.
I20260807 03:39:50.698756 22814 ts_tablet_manager.cc:1397] T 6b66966247c147308a4b8488557e0bdf P 5c9b5062bdfa46af9396f1a85025a331: Time spent bootstrapping tablet: real 1.088s user 0.040s sys 0.025s
I20260807 03:39:50.707080 18248 leader_election.cc:302] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:50.670229 22881 raft_consensus.cc:3037] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:50.717712 18075 raft_consensus.cc:3037] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:39:50.720436 18180 raft_consensus.cc:1226] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:50.728173 18075 raft_consensus.cc:1261] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 03:39:50.739951 18248 leader_election.cc:302] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
W20260807 03:39:50.740728 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 66224us
I20260807 03:39:50.747521 22931 raft_consensus.cc:2726] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260807 03:39:50.749020 18248 leader_election.cc:302] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:50.754150 18248 leader_election.cc:302] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:50.755844 18071 raft_consensus.cc:2454] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 6.
I20260807 03:39:50.739951 22426 raft_consensus.cc:2726] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260807 03:39:50.765059 18140 leader_election.cc:302] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:50.766034 22863 consensus_queue.cc:1046] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 03:39:50.767741 22863 raft_consensus.cc:2726] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260807 03:39:50.768404 22863 raft_consensus.cc:3032] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 03:39:50.768743 22863 raft_consensus.cc:729] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
W20260807 03:39:50.772960 22940 log.cc:924] Time spent T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.057s user 0.001s sys 0.000s
I20260807 03:39:50.773743 22658 consensus_queue.cc:1035] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:39:50.777246 22774 consensus_queue.cc:1046] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 03:39:50.777953 18075 raft_consensus.cc:1261] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20260807 03:39:50.778316 18075 pending_rounds.cc:76] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d: Aborting all ops after (but not including) 2
I20260807 03:39:50.778513 18075 pending_rounds.cc:97] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260807 03:39:50.778720 18075 raft_consensus.cc:2866] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 03:39:50.780648 22897 raft_consensus.cc:3032] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 03:39:50.781060 22897 raft_consensus.cc:729] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:50.758950 22931 raft_consensus.cc:2726] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 03:39:50.794847 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:39:50.785724 22427 raft_consensus.cc:2726] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 03:39:50.837195 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:39:50.836866 19240 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 56833.8us
I20260807 03:39:50.843199 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:50.843350 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:50.844018 18075 raft_consensus.cc:2379] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 7: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:50.782027 22897 consensus_queue.cc:260] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:50.845628 22897 raft_consensus.cc:3037] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:50.769570 22863 consensus_queue.cc:260] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:50.849084 17915 catalog_manager.cc:5495] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:50.850605 17915 catalog_manager.cc:5495] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:50.851572 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:50.852654 22934 consensus_queue.cc:1046] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.026s
I20260807 03:39:50.853253 22863 raft_consensus.cc:3037] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:50.856559 18184 raft_consensus.cc:2379] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 7: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:50.856607 22814 raft_consensus.cc:348] T 6b66966247c147308a4b8488557e0bdf P 5c9b5062bdfa46af9396f1a85025a331 [term 8 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:50.858031 22814 raft_consensus.cc:729] T 6b66966247c147308a4b8488557e0bdf P 5c9b5062bdfa46af9396f1a85025a331 [term 8 LEARNER]: Becoming Follower/Learner. State: Replica: 5c9b5062bdfa46af9396f1a85025a331, State: Initialized, Role: LEARNER
I20260807 03:39:50.858664 22814 consensus_queue.cc:260] T 6b66966247c147308a4b8488557e0bdf P 5c9b5062bdfa46af9396f1a85025a331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:39:50.851671 22827 raft_consensus.cc:482] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
W20260807 03:39:50.864578 22935 log.cc:924] Time spent T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.172s user 0.000s sys 0.003s
I20260807 03:39:50.865087 18184 raft_consensus.cc:1261] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260807 03:39:50.887151 22814 ts_tablet_manager.cc:1428] T 6b66966247c147308a4b8488557e0bdf P 5c9b5062bdfa46af9396f1a85025a331: Time spent starting tablet: real 0.188s user 0.008s sys 0.001s
I20260807 03:39:50.866133 22932 raft_consensus.cc:3032] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Stepping down as leader of term 5
I20260807 03:39:50.887544 22932 raft_consensus.cc:729] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:50.888386 22932 consensus_queue.cc:260] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:50.889586 22932 raft_consensus.cc:3037] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:50.837239 22862 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:55748 (request call id 6) took 4919 ms. Trace:
I20260807 03:39:50.870683 22132 consensus_queue.cc:1035] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260807 03:39:50.890442 22862 rpcz_store.cc:276] 0807 03:39:45.909066 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:45.909236 (+ 170us) service_pool.cc:227] Handling call
0807 03:39:46.118376 (+209140us) log.cc:842] Serialized 513 byte log entry
0807 03:39:47.419168 (+1300792us) spinlock_profiling.cc:243] Waited 22.3 ms on lock 0x7b5400080320. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe992150a4f 00007fe992139545 00007fe99213e771 00007fe99213e6f9 00007fe99213e689 00007fe99213e651 00007fe99213d81d 00007fe9984fb204 00007fe9984fb039 00007fe98f2976b6
0807 03:39:50.798727 (+3379559us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3478,"raft.queue_time_us":303990,"raft.run_cpu_time_us":16586,"raft.run_wall_time_us":811260,"spinlock_wait_cycles":51309440,"thread_start_us":745047,"threads_started":4,"wal-append.queue_time_us":238002,"wal-append.run_cpu_time_us":3890,"wal-append.run_wall_time_us":1520088}
W20260807 03:39:50.871145 22933 log.cc:924] Time spent T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.181s user 0.001s sys 0.000s
I20260807 03:39:50.829959 22476 consensus_queue.cc:1035] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20260807 03:39:50.825661 22916 raft_consensus.cc:504] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:50.873582 22934 raft_consensus.cc:3032] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 LEADER]: Stepping down as leader of term 6
I20260807 03:39:50.873258 18185 raft_consensus.cc:2932] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Committing config change with OpId 8.7: config changed from index -1 to 7, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:39:50.874332 18027 leader_election.cc:302] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:39:50.876535 18180 raft_consensus.cc:1261] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260807 03:39:50.885391 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:50.886273 19295 raft_consensus.cc:3037] T 6b66966247c147308a4b8488557e0bdf P 5c9b5062bdfa46af9396f1a85025a331 [term 8 LEARNER]: Advancing to term 9
I20260807 03:39:50.886722 22827 raft_consensus.cc:504] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:50.861704 22888 raft_consensus.cc:2932] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 9 LEADER]: Committing config change with OpId 8.7: config changed from index -1 to 7, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:39:50.906816 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:50.913944 19240 leader_election.cc:302] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:50.908249 22739 raft_consensus.cc:2726] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:50.921279 18075 raft_consensus.cc:1261] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260807 03:39:50.906123 22934 raft_consensus.cc:729] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:50.923705 22827 leader_election.cc:288] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:50.925711 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:50.915925 22832 raft_consensus.cc:2726] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260807 03:39:50.933109 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 58886.8us
I20260807 03:39:50.906733 22774 raft_consensus.cc:482] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:50.940722 22774 raft_consensus.cc:504] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:50.928357 22934 consensus_queue.cc:260] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:50.921554 18309 consensus_queue.cc:237] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:39:50.936973 19295 raft_consensus.cc:1261] T 6b66966247c147308a4b8488557e0bdf P 5c9b5062bdfa46af9396f1a85025a331 [term 9 LEARNER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260807 03:39:50.933466 18184 raft_consensus.cc:1261] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260807 03:39:50.906517 22781 consensus_queue.cc:1035] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.034s
I20260807 03:39:50.948453 18075 raft_consensus.cc:3032] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 8 LEADER]: Stepping down as leader of term 8
I20260807 03:39:50.948858 18075 raft_consensus.cc:729] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:50.955008 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:51.007920 18027 leader_election.cc:302] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:51.012506 18069 raft_consensus.cc:1261] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260807 03:39:50.946170 22774 leader_election.cc:288] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:51.012856 18075 consensus_queue.cc:260] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:39:51.023669 18075 raft_consensus.cc:3037] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:50.954735 22832 raft_consensus.cc:482] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:50.938802 22532 consensus_queue.cc:1035] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.074s
I20260807 03:39:51.010792 22842 raft_consensus.cc:482] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5c9b5062bdfa46af9396f1a85025a331)
I20260807 03:39:51.044975 18027 leader_election.cc:302] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:51.048847 18075 raft_consensus.cc:1261] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260807 03:39:50.955196 22931 consensus_queue.cc:1035] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.077s
I20260807 03:39:51.049530 18305 raft_consensus.cc:1261] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 03:39:51.053603 22132 raft_consensus.cc:2781] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:39:51.054839 18293 raft_consensus.cc:1261] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 03:39:51.057976 22132 raft_consensus.cc:482] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:51.058331 22132 raft_consensus.cc:3037] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:51.028261 22882 raft_consensus.cc:482] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:51.056697 18027 leader_election.cc:302] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:50.949555 22847 consensus_queue.cc:1035] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.081s
I20260807 03:39:51.036178 22934 raft_consensus.cc:3037] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:51.038515 22739 raft_consensus.cc:2726] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260807 03:39:51.046348 18094 tablet_copy_service.cc:342] P d9716e2bf06442f5b3b0e81087c53c4d: Request end of tablet copy session 5c9b5062bdfa46af9396f1a85025a331-6b66966247c147308a4b8488557e0bdf received from {username='slave'} at 127.0.0.1:58950
I20260807 03:39:51.068395 18075 raft_consensus.cc:1261] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260807 03:39:51.065663 18185 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
mode: GRACEFUL
new_leader_uuid: "66a36e1ac8e945e893668e282d394c49"
from {username='slave'} at 127.0.0.1:34758
I20260807 03:39:51.073261 18185 raft_consensus.cc:595] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Received request to transfer leadership to TS 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:51.073715 18185 raft_consensus.cc:603] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 03:39:51.076396 18094 tablet_copy_service.cc:433] P d9716e2bf06442f5b3b0e81087c53c4d: ending tablet copy session 5c9b5062bdfa46af9396f1a85025a331-6b66966247c147308a4b8488557e0bdf on tablet 6b66966247c147308a4b8488557e0bdf with peer 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:39:51.031603 22832 raft_consensus.cc:504] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
W20260807 03:39:51.090338 22945 log.cc:924] Time spent T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20260807 03:39:51.093657 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:51.095335 22916 leader_election.cc:288] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:51.052245 22842 raft_consensus.cc:504] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:51.102772 22842 leader_election.cc:288] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:51.103938 18027 leader_election.cc:302] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:51.105551 18027 leader_election.cc:302] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:51.106170 22857 raft_consensus.cc:2726] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 03:39:51.106453 22932 raft_consensus.cc:2726] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260807 03:39:51.107359 18248 leader_election.cc:302] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:51.107909 18027 leader_election.cc:302] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:51.110387 22866 raft_consensus.cc:2726] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260807 03:39:51.112166 22932 raft_consensus.cc:2726] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 03:39:51.113121 22841 raft_consensus.cc:482] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:51.080570 22902 raft_consensus.cc:2781] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 03:39:51.113711 22841 raft_consensus.cc:504] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:51.114542 22902 raft_consensus.cc:482] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:51.114971 22902 raft_consensus.cc:3037] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:51.116344 18027 leader_election.cc:302] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:51.118470 22132 raft_consensus.cc:504] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:51.120805 22132 leader_election.cc:288] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:51.121505 22932 raft_consensus.cc:2726] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:51.123782 19295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:51.125818 18071 raft_consensus.cc:1261] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 03:39:51.125952 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:51.127092 18248 leader_election.cc:302] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:51.127787 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:51.128172 18248 leader_election.cc:302] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:51.064437 22882 raft_consensus.cc:504] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:51.134516 22841 leader_election.cc:288] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:51.135248 22841 consensus_queue.cc:1035] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 03:39:51.136746 22426 raft_consensus.cc:3037] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:51.137982 18140 leader_election.cc:302] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:51.140021 22869 raft_consensus.cc:482] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:51.140553 22869 raft_consensus.cc:504] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:51.140895 22827 consensus_queue.cc:1035] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:39:51.141244 22863 raft_consensus.cc:2726] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260807 03:39:51.142313 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 79207.5us
I20260807 03:39:51.142573 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:51.142947 18141 leader_election.cc:302] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:51.143151 18185 raft_consensus.cc:2454] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 6.
I20260807 03:39:51.143570 22530 consensus_queue.cc:1035] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:39:51.144378 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:51.145298 22897 raft_consensus.cc:2726] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 03:39:51.145478 22530 consensus_queue.cc:1035] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:39:51.148597 18305 raft_consensus.cc:1261] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260807 03:39:51.148723 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:51.149506 18312 raft_consensus.cc:2379] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 6: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:51.152685 18183 raft_consensus.cc:1261] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 13 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260807 03:39:51.152637 22426 raft_consensus.cc:2726] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:51.152994 22257 consensus_queue.cc:1035] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:39:51.154472 19295 raft_consensus.cc:2932] T 6b66966247c147308a4b8488557e0bdf P 5c9b5062bdfa46af9396f1a85025a331 [term 9 LEARNER]: Committing config change with OpId 8.7: config changed from index -1 to 7, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:39:51.157800 22882 leader_election.cc:288] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:51.131831 22847 raft_consensus.cc:3037] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:51.160507 19240 leader_election.cc:302] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331; no voters:
I20260807 03:39:51.161324 22901 raft_consensus.cc:2781] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:39:51.161666 22901 raft_consensus.cc:482] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:51.161943 22901 raft_consensus.cc:3037] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:51.167598 22426 consensus_queue.cc:1035] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260807 03:39:51.167958 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:51.169885 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:51.170678 18190 raft_consensus.cc:2454] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 7.
W20260807 03:39:51.176499 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 60254.8us
I20260807 03:39:51.177465 18027 leader_election.cc:302] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:51.178689 18249 leader_election.cc:302] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:51.211159 18069 raft_consensus.cc:3037] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:51.223910 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:51.224558 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:51.221684 18248 leader_election.cc:302] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:51.225132 18082 raft_consensus.cc:2454] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 7.
I20260807 03:39:51.224561 18081 raft_consensus.cc:2454] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 6.
W20260807 03:39:51.218353 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 43685.3us
W20260807 03:39:51.228796 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 51441.8us
I20260807 03:39:51.142586 22869 leader_election.cc:288] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:51.232838 18140 leader_election.cc:302] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:51.233420 18081 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7"
dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
mode: GRACEFUL
new_leader_uuid: "3c4bceb836db4536a0f255fdf86abc02"
from {username='slave'} at 127.0.0.1:55742
I20260807 03:39:51.233954 18081 raft_consensus.cc:595] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 LEADER]: Received request to transfer leadership to TS 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:51.234026 18140 leader_election.cc:302] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:51.235424 18069 raft_consensus.cc:1261] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260807 03:39:51.236541 22658 raft_consensus.cc:2781] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:39:51.236928 22658 raft_consensus.cc:482] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:51.237300 22658 raft_consensus.cc:3037] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:51.180558 22882 raft_consensus.cc:482] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:51.169684 22832 leader_election.cc:288] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:51.208640 22838 raft_consensus.cc:482] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:51.244071 22882 raft_consensus.cc:504] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:51.244796 22838 raft_consensus.cc:504] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:51.246297 22871 raft_consensus.cc:482] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:51.246781 22838 leader_election.cc:288] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:51.246806 22871 raft_consensus.cc:504] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:51.247809 19292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:51.237569 22896 consensus_queue.cc:1035] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.006s
I20260807 03:39:51.248459 19292 raft_consensus.cc:2454] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 7.
I20260807 03:39:51.212428 22257 raft_consensus.cc:482] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:51.249248 22257 raft_consensus.cc:504] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:51.251567 22257 leader_election.cc:288] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:51.253002 19292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:51.253829 19292 raft_consensus.cc:2379] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 7: Already voted for candidate 5c9b5062bdfa46af9396f1a85025a331 in this term.
I20260807 03:39:51.254388 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:51.255821 18297 raft_consensus.cc:2454] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 7.
I20260807 03:39:51.178292 22426 consensus_queue.cc:1035] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.064s
I20260807 03:39:51.231980 22932 raft_consensus.cc:2726] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260807 03:39:51.238009 22427 raft_consensus.cc:2781] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 03:39:51.209540 22841 consensus_queue.cc:1035] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.044s
I20260807 03:39:51.262547 18027 leader_election.cc:302] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:51.225895 22842 consensus_queue.cc:1035] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20260807 03:39:51.281232 22132 raft_consensus.cc:482] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:51.281704 22132 raft_consensus.cc:504] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:51.272181 18027 leader_election.cc:302] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:39:51.260363 22931 raft_consensus.cc:2726] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260807 03:39:51.294246 18081 raft_consensus.cc:1261] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260807 03:39:51.297535 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:51.298763 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:51.301662 18190 raft_consensus.cc:1261] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260807 03:39:51.303315 18027 leader_election.cc:302] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:51.304817 18027 leader_election.cc:302] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:51.306877 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:51.299495 18184 raft_consensus.cc:2454] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 6.
I20260807 03:39:51.260346 22862 consensus_queue.cc:1046] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.034s
I20260807 03:39:51.270388 22427 raft_consensus.cc:482] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:51.315769 22132 leader_election.cc:288] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:51.292613 22929 raft_consensus.cc:2726] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:51.292762 22827 raft_consensus.cc:2781] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 03:39:51.296671 22932 raft_consensus.cc:2726] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260807 03:39:51.334823 18027 leader_election.cc:302] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:51.313060 22857 raft_consensus.cc:2726] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260807 03:39:51.336983 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:51.338829 18071 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "60437eed8852484c88e84993e376670e"
dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
mode: GRACEFUL
new_leader_uuid: "3c4bceb836db4536a0f255fdf86abc02"
from {username='slave'} at 127.0.0.1:55742
I20260807 03:39:51.340909 18071 raft_consensus.cc:595] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 7 LEADER]: Received request to transfer leadership to TS 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:51.311882 22776 raft_consensus.cc:2726] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260807 03:39:51.319087 22427 raft_consensus.cc:3037] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:51.319852 22869 raft_consensus.cc:482] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:51.317974 22886 raft_consensus.cc:482] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:51.340899 18069 raft_consensus.cc:2454] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 7.
I20260807 03:39:51.358624 18140 leader_election.cc:302] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:51.314739 22902 raft_consensus.cc:504] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:51.362339 18027 leader_election.cc:302] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
W20260807 03:39:51.368924 19240 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 86147.9us
I20260807 03:39:51.321486 22931 consensus_queue.cc:1035] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20260807 03:39:51.349838 22932 raft_consensus.cc:2726] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:51.373148 18246 leader_election.cc:302] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:51.346954 22882 leader_election.cc:288] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:51.380264 18075 raft_consensus.cc:1226] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 7. Current term is 8. Ops: []
I20260807 03:39:51.343302 22929 consensus_queue.cc:1046] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20260807 03:39:51.386444 18027 leader_election.cc:302] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:51.371538 22530 raft_consensus.cc:2781] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260807 03:39:51.390311 18246 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.132:41381, user_credentials={real_user=slave}} blocked reactor thread for 137151us
I20260807 03:39:51.389113 18069 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf"
dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
mode: GRACEFUL
new_leader_uuid: "66a36e1ac8e945e893668e282d394c49"
from {username='slave'} at 127.0.0.1:55742
I20260807 03:39:51.391283 18069 raft_consensus.cc:595] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Received request to transfer leadership to TS 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:51.391640 18069 raft_consensus.cc:603] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 03:39:51.391667 18141 leader_election.cc:302] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:39:51.390807 22530 raft_consensus.cc:482] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:51.395644 22530 raft_consensus.cc:3037] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Advancing to term 8
W20260807 03:39:51.399307 22959 log.cc:924] Time spent T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
I20260807 03:39:51.379324 22441 raft_consensus.cc:482] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:51.377846 22862 raft_consensus.cc:2726] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:51.353018 22869 raft_consensus.cc:504] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:51.358234 22886 raft_consensus.cc:504] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:51.403328 22132 raft_consensus.cc:3032] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 LEADER]: Stepping down as leader of term 4
I20260807 03:39:51.403748 22132 raft_consensus.cc:729] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:51.384793 22888 raft_consensus.cc:482] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:51.406652 19292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
W20260807 03:39:51.410295 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 107176us
I20260807 03:39:51.394279 22842 raft_consensus.cc:2781] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 03:39:51.412498 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:51.394603 22532 raft_consensus.cc:482] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:51.422595 18248 leader_election.cc:302] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:51.417774 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:51.425451 18309 raft_consensus.cc:2454] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 6.
I20260807 03:39:51.419755 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:51.426309 18312 raft_consensus.cc:2454] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 7.
I20260807 03:39:51.388969 22776 consensus_queue.cc:1046] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.025s
I20260807 03:39:51.422863 18183 raft_consensus.cc:2454] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 6.
I20260807 03:39:51.414084 22827 raft_consensus.cc:482] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:51.400064 22847 raft_consensus.cc:2726] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260807 03:39:51.434895 22963 log.cc:924] Time spent T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260807 03:39:51.394923 22871 leader_election.cc:288] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:51.435916 18189 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8c98c1d422654739af06653a512f7499"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
mode: GRACEFUL
new_leader_uuid: "66a36e1ac8e945e893668e282d394c49"
from {username='slave'} at 127.0.0.1:34758
I20260807 03:39:51.436275 18071 raft_consensus.cc:1226] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 6. Current term is 7. Ops: []
W20260807 03:39:51.436930 22960 log.cc:924] Time spent T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20260807 03:39:51.441625 22984 raft_consensus.cc:482] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:51.442123 22984 raft_consensus.cc:504] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:51.444377 22827 raft_consensus.cc:3037] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Advancing to term 8
W20260807 03:39:51.444653 22967 log.cc:924] Time spent T 6b66966247c147308a4b8488557e0bdf P 5c9b5062bdfa46af9396f1a85025a331: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260807 03:39:51.445941 22975 consensus_queue.cc:1035] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 03:39:51.402125 22857 raft_consensus.cc:2726] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 03:39:51.447189 22975 raft_consensus.cc:2726] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:51.423566 22842 raft_consensus.cc:482] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:51.447678 22842 raft_consensus.cc:3037] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:51.401099 22882 raft_consensus.cc:2781] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260807 03:39:51.434172 22961 log.cc:924] Time spent T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20260807 03:39:51.452257 22739 consensus_queue.cc:1046] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 03:39:51.453401 22882 raft_consensus.cc:482] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:51.453577 22739 raft_consensus.cc:3032] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 LEADER]: Stepping down as leader of term 4
I20260807 03:39:51.454087 22739 raft_consensus.cc:729] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:51.453836 22882 raft_consensus.cc:3037] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:51.400826 22520 raft_consensus.cc:482] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:51.455003 22132 consensus_queue.cc:260] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:51.455994 22132 raft_consensus.cc:3037] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:51.444020 22984 leader_election.cc:288] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
W20260807 03:39:51.469933 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 44248.2us
I20260807 03:39:51.470592 18027 leader_election.cc:302] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:51.471441 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:51.472083 18075 raft_consensus.cc:2454] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 6.
I20260807 03:39:51.455178 22739 consensus_queue.cc:260] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:51.481351 22869 leader_election.cc:288] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:51.482833 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:51.483458 18082 raft_consensus.cc:2454] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 9.
I20260807 03:39:51.483513 22886 leader_election.cc:288] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:51.484702 22902 leader_election.cc:288] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:51.485337 22520 raft_consensus.cc:504] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:51.487438 22520 leader_election.cc:288] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:51.495725 18140 leader_election.cc:302] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:51.400414 22441 raft_consensus.cc:504] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:51.536768 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 66229.2us
W20260807 03:39:51.537350 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 47878.1us
I20260807 03:39:51.538123 18027 leader_election.cc:398] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 12 pre-election: Vote denied by peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057) with higher term. Message: Invalid argument: T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 12. Current term is 13.
I20260807 03:39:51.538414 18027 leader_election.cc:401] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 12 pre-election: Cancelling election due to peer responding with higher term
I20260807 03:39:51.540652 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:51.541469 18070 raft_consensus.cc:2454] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
I20260807 03:39:51.420414 22888 raft_consensus.cc:504] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:51.549315 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:51.552747 19295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:51.425215 22532 raft_consensus.cc:504] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:51.553453 19295 raft_consensus.cc:2379] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 4: Already voted for candidate 5c9b5062bdfa46af9396f1a85025a331 in this term.
I20260807 03:39:51.554054 18248 leader_election.cc:302] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:51.540594 22857 raft_consensus.cc:2781] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election won for term 7
I20260807 03:39:51.545269 22869 raft_consensus.cc:2781] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260807 03:39:51.495198 22841 consensus_queue.cc:1046] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.058s
I20260807 03:39:51.549918 22986 consensus_queue.cc:1046] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.019s
I20260807 03:39:51.565873 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:51.566512 18293 raft_consensus.cc:2454] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 9.
I20260807 03:39:51.566959 22888 leader_election.cc:288] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:51.565038 22865 raft_consensus.cc:2726] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: Vote denied by peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057) with higher term. Message: Invalid argument: T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 12. Current term is 13.
I20260807 03:39:51.570922 18248 leader_election.cc:302] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:39:51.574334 22532 leader_election.cc:288] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:51.583343 22842 raft_consensus.cc:504] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:51.562569 22869 raft_consensus.cc:482] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:51.610374 22869 raft_consensus.cc:3037] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Advancing to term 10
I20260807 03:39:51.616587 22841 raft_consensus.cc:3032] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Stepping down as leader of term 7
I20260807 03:39:51.616979 22841 raft_consensus.cc:729] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:51.617249 22532 raft_consensus.cc:2781] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260807 03:39:51.617372 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 46539.7us
I20260807 03:39:51.617653 22532 raft_consensus.cc:482] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:51.618119 22532 raft_consensus.cc:3037] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:51.620577 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:51.621172 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:51.621205 18070 raft_consensus.cc:2454] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 8.
I20260807 03:39:51.621785 18071 raft_consensus.cc:2454] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 7.
I20260807 03:39:51.622891 18070 raft_consensus.cc:1261] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 13 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
W20260807 03:39:51.598774 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 60751.1us
I20260807 03:39:51.602116 22658 raft_consensus.cc:504] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:51.617694 22841 consensus_queue.cc:260] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:51.602536 22530 raft_consensus.cc:504] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:51.624820 18027 leader_election.cc:302] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:51.626856 22530 leader_election.cc:288] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:51.585052 22984 raft_consensus.cc:2781] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 03:39:51.607106 22882 raft_consensus.cc:504] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:51.608906 19295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:51.627394 22984 raft_consensus.cc:482] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:51.627812 22984 raft_consensus.cc:3037] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:51.620743 22871 raft_consensus.cc:1050] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49: attempting to promote NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 to VOTER
I20260807 03:39:51.628655 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:51.629276 18071 raft_consensus.cc:3037] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:51.627873 19295 raft_consensus.cc:2379] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 7: Already voted for candidate 5c9b5062bdfa46af9396f1a85025a331 in this term.
I20260807 03:39:51.629736 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:51.631161 18248 leader_election.cc:302] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:51.632092 22762 raft_consensus.cc:2781] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 03:39:51.632140 22882 leader_election.cc:288] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 election: Requested vote from peers 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:51.632500 22762 raft_consensus.cc:482] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:51.569847 22857 raft_consensus.cc:686] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:51.625005 22658 leader_election.cc:288] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:51.636132 22762 raft_consensus.cc:3037] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:51.638972 18183 raft_consensus.cc:1203] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Deduplicated request from leader. Original: 8.6->[8.7-8.7] Dedup: 8.7->[]
I20260807 03:39:51.639468 18183 raft_consensus.cc:1226] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 8. Current term is 9. Ops: [8.7-8.7]
I20260807 03:39:51.624846 22841 raft_consensus.cc:3037] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:51.633162 19295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331"
I20260807 03:39:51.641075 19295 raft_consensus.cc:2379] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 7: Already voted for candidate 5c9b5062bdfa46af9396f1a85025a331 in this term.
I20260807 03:39:51.593063 22441 leader_election.cc:288] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:51.634676 22929 raft_consensus.cc:3037] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:51.635517 22842 leader_election.cc:288] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
W20260807 03:39:51.635656 22983 log.cc:924] Time spent T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20260807 03:39:51.641091 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:51.633311 18248 leader_election.cc:302] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:51.643489 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:51.640589 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:51.650183 18249 leader_election.cc:302] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:51.657471 18191 raft_consensus.cc:1226] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 7. Current term is 8. Ops: []
I20260807 03:39:51.657802 19297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331"
I20260807 03:39:51.658335 19297 raft_consensus.cc:3037] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:51.659044 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:51.660423 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:51.648844 18027 leader_election.cc:302] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:51.641640 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:51.661357 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:51.661775 18180 raft_consensus.cc:2454] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 9.
I20260807 03:39:51.643707 18070 raft_consensus.cc:2454] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 6.
I20260807 03:39:51.662312 18027 leader_election.cc:302] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:51.663234 22132 raft_consensus.cc:2726] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 03:39:51.663478 18027 leader_election.cc:302] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:51.664253 22132 raft_consensus.cc:2726] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260807 03:39:51.664348 18140 leader_election.cc:302] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:51.669400 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:51.668411 18071 raft_consensus.cc:2454] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 8.
I20260807 03:39:51.668977 18193 raft_consensus.cc:2354] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 7. Current term is 8.
I20260807 03:39:51.672145 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:51.677896 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:51.675004 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:51.675395 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:51.683923 18185 raft_consensus.cc:1226] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 4. Current term is 5. Ops: []
I20260807 03:39:51.686357 18140 leader_election.cc:302] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:51.662039 22817 raft_consensus.cc:2781] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:39:51.668977 22847 raft_consensus.cc:2781] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260807 03:39:51.682783 22901 consensus_meta.cc:287] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331: Time spent flushing consensus metadata: real 0.520s user 0.008s sys 0.004s
I20260807 03:39:51.694823 22530 raft_consensus.cc:2781] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader election won for term 8
I20260807 03:39:51.695478 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:51.697616 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:51.703034 18185 raft_consensus.cc:2454] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 7.
I20260807 03:39:51.670190 22776 raft_consensus.cc:2726] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 03:39:51.704397 22909 raft_consensus.cc:482] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:51.705039 22909 raft_consensus.cc:504] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:51.707486 22909 leader_election.cc:288] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:51.709188 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:51.709831 18190 raft_consensus.cc:2454] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 8.
I20260807 03:39:51.709580 22427 raft_consensus.cc:504] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:51.710638 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:51.711524 22427 leader_election.cc:288] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:51.711674 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:51.714504 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:51.715020 18190 raft_consensus.cc:3037] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:51.712345 22901 raft_consensus.cc:504] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:51.690759 22897 raft_consensus.cc:2781] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 03:39:51.715378 18071 raft_consensus.cc:2454] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 3.
I20260807 03:39:51.718765 19297 raft_consensus.cc:2454] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 8.
I20260807 03:39:51.727783 22901 leader_election.cc:288] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 7 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:51.728874 19292 raft_consensus.cc:2379] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 7: Already voted for candidate 5c9b5062bdfa46af9396f1a85025a331 in this term.
I20260807 03:39:51.730625 22869 raft_consensus.cc:504] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:51.731611 22977 log.cc:924] Time spent T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.145s user 0.000s sys 0.000s
I20260807 03:39:51.701272 22827 raft_consensus.cc:504] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:51.732352 18183 raft_consensus.cc:2354] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 9. Current term is 10.
I20260807 03:39:51.723997 22530 raft_consensus.cc:686] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 8 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:51.734313 18189 raft_consensus.cc:595] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Received request to transfer leadership to TS 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:51.734660 18189 raft_consensus.cc:603] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 03:39:51.735110 19239 leader_election.cc:302] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:51.694562 22817 raft_consensus.cc:482] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:51.735662 22817 raft_consensus.cc:3037] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:51.736672 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:51.737428 18187 raft_consensus.cc:3037] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:51.700973 22847 raft_consensus.cc:482] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:51.740290 22847 raft_consensus.cc:3037] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:51.735925 22604 raft_consensus.cc:2781] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:51.740912 22604 raft_consensus.cc:482] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:51.741357 22604 raft_consensus.cc:3037] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:51.742833 22984 raft_consensus.cc:504] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:51.744724 22969 log.cc:924] Time spent T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.291s user 0.004s sys 0.000s
I20260807 03:39:51.677625 22781 consensus_queue.cc:1035] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.035s
W20260807 03:39:51.756847 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 69347.2us
I20260807 03:39:51.759044 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
W20260807 03:39:51.769804 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 104462us
I20260807 03:39:51.770501 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:51.770536 18027 leader_election.cc:302] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:51.770812 18187 raft_consensus.cc:2454] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 7.
I20260807 03:39:51.772774 19240 leader_election.cc:302] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331; no voters:
I20260807 03:39:51.748837 22827 leader_election.cc:288] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:51.776532 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:51.779327 22974 raft_consensus.cc:2781] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 03:39:51.779867 22974 raft_consensus.cc:686] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 7 LEADER]: Becoming Leader. State: Replica: 5c9b5062bdfa46af9396f1a85025a331, State: Running, Role: LEADER
I20260807 03:39:51.780676 22974 consensus_queue.cc:237] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:51.782908 22762 raft_consensus.cc:504] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:51.753266 22897 raft_consensus.cc:686] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:51.784528 22869 leader_election.cc:288] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:51.784261 22897 consensus_queue.cc:237] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:51.755304 22532 raft_consensus.cc:504] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:51.787494 22532 leader_election.cc:288] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:51.788424 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:51.767930 22257 raft_consensus.cc:482] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:51.789984 22257 raft_consensus.cc:504] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:51.792789 22776 raft_consensus.cc:2726] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 03:39:51.748034 22530 consensus_queue.cc:237] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:51.811555 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:51.812940 22817 raft_consensus.cc:504] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:51.815001 22817 leader_election.cc:288] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:51.816569 22257 leader_election.cc:288] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:51.816969 22762 leader_election.cc:288] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:51.818445 18311 raft_consensus.cc:2379] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 5c9b5062bdfa46af9396f1a85025a331 in current term 7: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:51.819887 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:51.820708 18082 raft_consensus.cc:3037] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Advancing to term 9
W20260807 03:39:51.824555 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 52525.7us
I20260807 03:39:51.838215 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:51.842193 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:51.842971 18185 raft_consensus.cc:2379] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 7: Already voted for candidate 5c9b5062bdfa46af9396f1a85025a331 in this term.
I20260807 03:39:51.751739 22857 consensus_queue.cc:237] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:51.844885 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:51.845053 18249 leader_election.cc:302] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:51.847991 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:51.849504 18193 raft_consensus.cc:3037] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:51.849730 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:51.850270 18070 raft_consensus.cc:3037] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Advancing to term 10
I20260807 03:39:51.865837 18071 raft_consensus.cc:2454] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
I20260807 03:39:51.867717 18027 leader_election.cc:302] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:51.853308 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:51.870187 18082 raft_consensus.cc:2454] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 9.
I20260807 03:39:51.815972 22847 raft_consensus.cc:504] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:51.863294 18070 raft_consensus.cc:2454] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 10.
I20260807 03:39:51.872540 18248 leader_election.cc:302] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:51.877795 18248 leader_election.cc:302] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:51.851320 22984 leader_election.cc:288] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:51.845599 22776 consensus_queue.cc:1046] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20260807 03:39:51.869272 22532 raft_consensus.cc:2726] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 03:39:51.871405 22975 raft_consensus.cc:482] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5c9b5062bdfa46af9396f1a85025a331)
I20260807 03:39:51.904469 22996 raft_consensus.cc:2781] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 03:39:51.905172 22997 raft_consensus.cc:482] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:51.881743 22986 raft_consensus.cc:2781] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader election won for term 9
I20260807 03:39:51.920727 18027 leader_election.cc:302] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:51.918849 22997 raft_consensus.cc:504] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:51.938465 18027 leader_election.cc:302] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:51.947535 22997 leader_election.cc:288] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:51.948185 23003 raft_consensus.cc:482] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:51.948637 22990 raft_consensus.cc:2726] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260807 03:39:51.948745 23003 raft_consensus.cc:504] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:51.950145 22990 raft_consensus.cc:2726] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 03:39:51.949927 22604 raft_consensus.cc:504] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:51.950927 23003 leader_election.cc:288] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:51.952037 22604 leader_election.cc:288] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:51.953529 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:51.954067 18073 raft_consensus.cc:3037] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
W20260807 03:39:51.956629 23007 log.cc:924] Time spent T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20260807 03:39:51.960940 22847 leader_election.cc:288] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
W20260807 03:39:51.948341 22992 log.cc:924] Time spent T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.097s user 0.001s sys 0.000s
I20260807 03:39:51.985339 18190 raft_consensus.cc:2454] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 8.
I20260807 03:39:51.986593 18193 raft_consensus.cc:2454] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 9.
I20260807 03:39:51.961647 18073 raft_consensus.cc:2454] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 4.
I20260807 03:39:51.969938 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
W20260807 03:39:51.991657 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 53693.3us
I20260807 03:39:51.992717 18027 leader_election.cc:302] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:51.994684 22932 raft_consensus.cc:3032] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 LEADER]: Stepping down as leader of term 7
I20260807 03:39:51.995110 22932 raft_consensus.cc:729] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:51.921128 22996 raft_consensus.cc:686] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:51.995932 22932 consensus_queue.cc:260] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:51.996253 22996 consensus_queue.cc:237] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:51.997257 22932 raft_consensus.cc:3037] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:51.988953 19239 leader_election.cc:302] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
W20260807 03:39:51.965071 23004 log.cc:924] Time spent T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.141s user 0.001s sys 0.000s
I20260807 03:39:52.004616 18140 leader_election.cc:302] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:52.008037 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:52.008492 18027 leader_election.cc:302] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
W20260807 03:39:52.006556 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 127495us
I20260807 03:39:51.964821 22986 raft_consensus.cc:686] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 9 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:52.011322 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:52.014863 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:52.007854 22865 raft_consensus.cc:2726] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:51.919005 22975 raft_consensus.cc:504] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:52.022145 22776 raft_consensus.cc:2726] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:52.012722 22891 raft_consensus.cc:2781] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 03:39:52.077802 22774 raft_consensus.cc:482] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:52.020628 22847 raft_consensus.cc:482] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:52.111821 18071 raft_consensus.cc:1226] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 6. Current term is 7. Ops: []
I20260807 03:39:52.274580 18249 leader_election.cc:302] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:52.278153 18249 leader_election.cc:302] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:52.277266 18073 raft_consensus.cc:2454] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 7.
W20260807 03:39:52.283144 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 274695us
I20260807 03:39:52.283859 18027 leader_election.cc:302] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:52.274947 22847 raft_consensus.cc:504] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:52.217706 22869 raft_consensus.cc:482] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:52.268667 23001 raft_consensus.cc:482] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:52.287623 22865 raft_consensus.cc:2726] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:52.289996 22427 raft_consensus.cc:482] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:52.297195 23017 raft_consensus.cc:979] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d: : Instructing follower 3c4bceb836db4536a0f255fdf86abc02 to start an election
I20260807 03:39:52.294054 22520 raft_consensus.cc:482] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
W20260807 03:39:52.290985 22929 consensus_meta.cc:287] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 0.631s user 0.012s sys 0.000s
I20260807 03:39:52.296033 22869 raft_consensus.cc:504] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:52.286265 22781 raft_consensus.cc:2726] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260807 03:39:52.094538 22774 raft_consensus.cc:504] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:52.275087 22993 consensus_queue.cc:1046] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.271s
I20260807 03:39:52.295814 18027 leader_election.cc:398] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Vote denied by peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057) with higher term. Message: Invalid argument: T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 7. Current term is 8.
I20260807 03:39:52.191876 18293 raft_consensus.cc:3037] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [term 2 FOLLOWER]: Advancing to term 4
I20260807 03:39:52.314342 22520 raft_consensus.cc:504] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:52.315006 22985 raft_consensus.cc:482] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:52.311904 22865 raft_consensus.cc:1067] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 LEADER]: Signalling peer 3c4bceb836db4536a0f255fdf86abc02 to start an election
I20260807 03:39:52.315572 22985 raft_consensus.cc:504] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:52.316622 22520 leader_election.cc:288] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:52.317555 22985 leader_election.cc:288] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:52.184072 23003 raft_consensus.cc:482] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:52.320719 18190 raft_consensus.cc:1226] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 7. Current term is 8. Ops: []
I20260807 03:39:52.289291 22866 raft_consensus.cc:482] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:52.320119 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:52.320027 22988 raft_consensus.cc:2932] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [term 6 LEADER]: Committing config change with OpId 6.5: config changed from index -1 to 5, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:39:52.289496 23015 raft_consensus.cc:2726] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260807 03:39:52.326542 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:52.327261 18193 raft_consensus.cc:3037] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Advancing to term 8
W20260807 03:39:52.327423 18246 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.132:41381, user_credentials={real_user=slave}} blocked reactor thread for 676929us
I20260807 03:39:52.327565 18071 raft_consensus.cc:1261] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260807 03:39:52.330420 18071 raft_consensus.cc:1261] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260807 03:39:52.331516 18310 raft_consensus.cc:1226] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 7. Current term is 8. Ops: []
I20260807 03:39:52.331588 18298 raft_consensus.cc:1261] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260807 03:39:52.332564 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:52.332916 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:52.333741 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:52.309366 22817 raft_consensus.cc:2781] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader election won for term 8
I20260807 03:39:52.345768 22774 leader_election.cc:288] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:52.346740 23022 log.cc:924] Time spent T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.085s user 0.001s sys 0.000s
I20260807 03:39:52.335682 18027 leader_election.cc:398] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 pre-election: Vote denied by peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057) with higher term. Message: Invalid argument: T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 9. Current term is 10.
I20260807 03:39:52.343804 22817 raft_consensus.cc:686] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 8 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:52.337131 18293 raft_consensus.cc:2454] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 4.
I20260807 03:39:52.339025 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:52.339713 18305 raft_consensus.cc:1226] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 6. Current term is 7. Ops: []
I20260807 03:39:52.340520 18300 raft_consensus.cc:1226] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:52.308542 23016 raft_consensus.cc:2781] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Leader election won for term 10
I20260807 03:39:52.336714 18183 raft_consensus.cc:2396] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260807 03:39:52.322113 22929 raft_consensus.cc:2726] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260807 03:39:52.410894 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 85340us
I20260807 03:39:52.341259 18309 raft_consensus.cc:1226] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 7. Current term is 8. Ops: []
I20260807 03:39:52.413101 19297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331"
I20260807 03:39:52.419581 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:52.423918 19297 raft_consensus.cc:2379] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 4: Already voted for candidate 5c9b5062bdfa46af9396f1a85025a331 in this term.
I20260807 03:39:52.321545 22869 leader_election.cc:288] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:52.425338 18249 leader_election.cc:302] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:52.297508 22427 raft_consensus.cc:504] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:52.425906 18072 raft_consensus.cc:3037] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:52.427165 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:52.419888 18191 raft_consensus.cc:2932] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Committing config change with OpId 6.5: config changed from index -1 to 5, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:39:52.428146 18188 raft_consensus.cc:2454] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:52.424278 18187 raft_consensus.cc:2454] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:52.430416 19297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331"
I20260807 03:39:52.430755 18248 leader_election.cc:302] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:52.218637 22986 consensus_queue.cc:237] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:52.445611 18071 raft_consensus.cc:2454] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 8.
W20260807 03:39:52.425426 23008 log.cc:924] Time spent T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.097s user 0.001s sys 0.000s
I20260807 03:39:52.415952 22257 raft_consensus.cc:482] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:52.302647 23001 raft_consensus.cc:504] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:52.436251 22520 consensus_queue.cc:1046] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.017s
I20260807 03:39:52.453117 22932 consensus_queue.cc:1035] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 03:39:52.452839 22817 consensus_queue.cc:237] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:52.444993 22990 raft_consensus.cc:482] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:52.336004 22866 raft_consensus.cc:504] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:52.455937 22987 raft_consensus.cc:482] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:52.458539 22984 consensus_queue.cc:1046] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 03:39:52.459838 23028 consensus_queue.cc:1035] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 03:39:52.426836 23016 raft_consensus.cc:686] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 10 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:52.461912 18298 raft_consensus.cc:2454] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 8.
I20260807 03:39:52.326347 23003 raft_consensus.cc:504] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:52.413779 22891 raft_consensus.cc:686] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 4 LEADER]: Becoming Leader. State: Replica: 5c9b5062bdfa46af9396f1a85025a331, State: Running, Role: LEADER
I20260807 03:39:52.483574 22866 leader_election.cc:288] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:52.483589 23003 leader_election.cc:288] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:52.484275 22891 consensus_queue.cc:237] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:52.450371 18193 raft_consensus.cc:2454] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 8.
I20260807 03:39:52.469353 18072 raft_consensus.cc:2454] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 8.
W20260807 03:39:52.472325 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 47070.1us
W20260807 03:39:52.473450 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 47347.1us
I20260807 03:39:52.486871 22441 consensus_queue.cc:1035] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 03:39:52.487300 22427 leader_election.cc:288] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:52.489336 23003 raft_consensus.cc:482] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:52.489812 23003 raft_consensus.cc:504] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:52.490470 22847 leader_election.cc:288] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:52.490638 18187 raft_consensus.cc:1261] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260807 03:39:52.491982 18187 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
from {username='slave'} at 127.0.0.1:44662
I20260807 03:39:52.492499 18187 raft_consensus.cc:482] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:39:52.492858 18187 raft_consensus.cc:3037] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:52.494644 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:52.470202 22975 leader_election.cc:288] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:52.478951 22990 raft_consensus.cc:504] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:52.496996 23001 leader_election.cc:288] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:52.497833 22987 raft_consensus.cc:504] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:52.493135 23003 leader_election.cc:288] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:52.471736 22685 raft_consensus.cc:482] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:52.502060 19297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331"
I20260807 03:39:52.500439 18187 raft_consensus.cc:504] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:52.473493 22257 raft_consensus.cc:504] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:52.506553 22987 leader_election.cc:288] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:52.571729 18189 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:34758 (request call id 32) took 1136 ms. Trace:
I20260807 03:39:52.503211 18191 raft_consensus.cc:1261] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Refusing update from remote peer 5c9b5062bdfa46af9396f1a85025a331: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260807 03:39:52.845978 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 59609.2us
W20260807 03:39:52.808733 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 378017us
I20260807 03:39:52.523799 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:52.497864 22990 leader_election.cc:288] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:52.833266 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:52.950027 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:52.833330 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:52.940263 19297 raft_consensus.cc:2379] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 7: Already voted for candidate 5c9b5062bdfa46af9396f1a85025a331 in this term.
I20260807 03:39:52.950917 18297 raft_consensus.cc:3037] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Advancing to term 10
I20260807 03:39:52.767200 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:52.574589 17915 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39492 (request call id 225) took 1829 ms. Trace:
I20260807 03:39:52.584272 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:52.545006 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 112590us
I20260807 03:39:52.522919 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:52.962659 18079 raft_consensus.cc:2454] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 5.
I20260807 03:39:52.949380 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
W20260807 03:39:52.534983 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 48464.6us
I20260807 03:39:52.961341 18187 leader_election.cc:288] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:52.928824 22984 raft_consensus.cc:482] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:52.965147 22984 raft_consensus.cc:504] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:52.966912 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:52.967131 22984 leader_election.cc:288] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:52.959892 18298 raft_consensus.cc:2454] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 7.
I20260807 03:39:52.963660 18309 raft_consensus.cc:2379] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 6: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:52.960346 18305 raft_consensus.cc:3037] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:52.957333 17915 rpcz_store.cc:276] 0807 03:39:50.745523 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:50.745771 (+ 248us) service_pool.cc:227] Handling call
0807 03:39:52.574499 (+1828728us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 03:39:51.099028 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 03:39:51.099634 (+ 606us) write_op.cc:432] Acquiring schema lock in shared mode
0807 03:39:51.099695 (+ 61us) write_op.cc:435] Acquired schema lock
0807 03:39:51.099721 (+ 26us) tablet.cc:660] Decoding operations
0807 03:39:51.131155 (+ 31434us) write_op.cc:620] Acquiring the partition lock for write op
0807 03:39:51.132736 (+ 1581us) write_op.cc:641] Partition lock acquired for write op
0807 03:39:51.133747 (+ 1011us) tablet.cc:683] Acquiring locks for 2 operations
0807 03:39:51.168356 (+ 34609us) tablet.cc:699] Row locks acquired
0807 03:39:51.168377 (+ 21us) write_op.cc:260] PREPARE: finished
0807 03:39:51.168510 (+ 133us) write_op.cc:270] Start()
0807 03:39:51.168628 (+ 118us) write_op.cc:276] Timestamp: P: 1786073991168468 usec, L: 0
0807 03:39:51.168647 (+ 19us) op_driver.cc:331] REPLICATION: starting
0807 03:39:51.178155 (+ 9508us) log.cc:842] Serialized 935 byte log entry
0807 03:39:51.391566 (+213411us) op_driver.cc:444] REPLICATION: finished
0807 03:39:51.641436 (+249870us) write_op.cc:301] APPLY: starting
0807 03:39:51.641537 (+ 101us) tablet.cc:1365] starting BulkCheckPresence
0807 03:39:51.641671 (+ 134us) tablet.cc:1368] finished BulkCheckPresence
0807 03:39:51.671842 (+ 30171us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 03:39:51.673406 (+ 1564us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 03:39:51.673663 (+ 257us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0807 03:39:51.673714 (+ 51us) write_op.cc:312] APPLY: finished
0807 03:39:51.673938 (+ 224us) log.cc:842] Serialized 29 byte log entry
0807 03:39:51.674116 (+ 178us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:39:51.674219 (+ 103us) write_op.cc:454] Released schema lock
0807 03:39:51.674272 (+ 53us) write_op.cc:341] FINISH: Updating metrics
0807 03:39:51.674567 (+ 295us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:39:51.674608 (+ 41us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":239207,"apply.run_cpu_time_us":2669,"apply.run_wall_time_us":33439,"num_ops":2,"prepare.queue_time_us":203096,"prepare.run_cpu_time_us":3256,"prepare.run_wall_time_us":113705,"replication_time_us":218159,"thread_start_us":281760,"threads_started":2}]]}
I20260807 03:39:52.961179 18070 raft_consensus.cc:2454] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 8.
I20260807 03:39:52.479542 22985 raft_consensus.cc:2726] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260807 03:39:52.480224 22520 raft_consensus.cc:2781] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 03:39:52.526736 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:52.977509 18249 leader_election.cc:302] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:52.973448 18027 leader_election.cc:302] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:52.960433 18189 rpcz_store.cc:276] 0807 03:39:51.435654 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:51.435788 (+ 134us) service_pool.cc:227] Handling call
0807 03:39:52.571535 (+1135747us) spinlock_profiling.cc:243] Waited 298 ms on lock 0x7b5400195854. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921b9bc3 00007fe99710381a 00007fe9907f6d04 00007fe9907f6c92 00007fe9907f6bc1 00007fe9907f6b3c 00007fe9907f5df2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:52.571643 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":684694400}
I20260807 03:39:52.578274 17918 catalog_manager.cc:5495] T 6b66966247c147308a4b8488557e0bdf P 5c9b5062bdfa46af9396f1a85025a331 reported cstate change: config changed from index -1 to 7, term changed from 8 to 9, leader changed from d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131), NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New cstate: current_term: 9 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:39:52.983404 22853 raft_consensus.cc:482] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:52.983971 22817 raft_consensus.cc:482] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:52.984365 18249 leader_election.cc:302] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:52.984462 22817 raft_consensus.cc:504] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:52.985646 22427 raft_consensus.cc:2781] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:52.985718 18249 leader_election.cc:302] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:52.986114 22427 raft_consensus.cc:482] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:52.986539 22427 raft_consensus.cc:3037] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:52.983979 22853 raft_consensus.cc:504] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:52.988265 22866 raft_consensus.cc:2781] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:52.988611 22866 raft_consensus.cc:482] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:52.988838 22853 leader_election.cc:288] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:52.989614 22853 raft_consensus.cc:2781] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader election won for term 8
I20260807 03:39:52.989765 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:52.988971 22866 raft_consensus.cc:3037] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:52.990196 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:52.986539 22817 leader_election.cc:288] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:52.990931 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:52.990154 22853 raft_consensus.cc:686] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 8 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:52.992308 18183 raft_consensus.cc:1261] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260807 03:39:52.993420 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:52.993973 18187 raft_consensus.cc:2454] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 8.
I20260807 03:39:52.990517 18191 raft_consensus.cc:2454] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 9.
I20260807 03:39:52.994937 22996 raft_consensus.cc:482] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:52.994599 22427 raft_consensus.cc:504] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:52.995406 22996 raft_consensus.cc:504] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:52.996534 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:52.997435 22427 leader_election.cc:288] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:52.997818 22996 leader_election.cc:288] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:52.980931 18300 raft_consensus.cc:2454] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 8.
I20260807 03:39:52.781225 23017 raft_consensus.cc:482] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:53.003497 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:52.799719 22865 raft_consensus.cc:482] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:52.781618 22441 raft_consensus.cc:482] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:52.850526 22658 raft_consensus.cc:482] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:52.531154 23001 raft_consensus.cc:482] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:52.531687 23003 raft_consensus.cc:482] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:53.033154 23025 raft_consensus.cc:482] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:53.033588 22441 raft_consensus.cc:504] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:52.555428 23016 consensus_queue.cc:237] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:52.951129 22685 raft_consensus.cc:504] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:53.036845 22685 leader_election.cc:288] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:52.969002 22888 raft_consensus.cc:3032] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 6 LEADER]: Stepping down as leader of term 6
I20260807 03:39:53.039280 22658 raft_consensus.cc:504] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:53.037606 22888 raft_consensus.cc:729] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:52.768224 22530 raft_consensus.cc:482] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:53.015177 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:53.041745 18187 raft_consensus.cc:2454] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:53.021323 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260807 03:39:53.043272 23051 raft_consensus.cc:482] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:53.043751 23051 raft_consensus.cc:504] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:53.044402 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:39:53.045238 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:39:53.046578 23046 consensus_queue.cc:1035] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260807 03:39:53.047821 23001 raft_consensus.cc:504] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:53.048964 18188 raft_consensus.cc:2454] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:53.049926 23001 leader_election.cc:288] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:53.025859 23017 raft_consensus.cc:504] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:53.052915 23037 raft_consensus.cc:482] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:53.053074 23017 leader_election.cc:288] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:53.052762 22530 raft_consensus.cc:504] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:53.053421 23037 raft_consensus.cc:504] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:53.055397 23037 leader_election.cc:288] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:53.037668 18249 leader_election.cc:302] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:53.057049 18249 leader_election.cc:302] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:53.014955 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:53.020591 18193 raft_consensus.cc:1226] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:53.016770 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:53.058514 18309 raft_consensus.cc:2454] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 8.
I20260807 03:39:53.018770 22257 leader_election.cc:288] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:53.032830 23003 raft_consensus.cc:504] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:53.033721 23025 raft_consensus.cc:504] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:53.014497 18191 raft_consensus.cc:2454] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:53.041147 22739 raft_consensus.cc:3037] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:53.015239 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:53.012856 18297 raft_consensus.cc:2454] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 10.
I20260807 03:39:53.076278 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:53.077349 18297 raft_consensus.cc:2454] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 8.
I20260807 03:39:53.059921 18312 raft_consensus.cc:2454] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 8.
I20260807 03:39:53.083550 18249 leader_election.cc:302] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:53.054256 23051 leader_election.cc:288] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:53.089179 18246 consensus_peers.cc:487] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 -> Peer 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381): Couldn't send request to peer 5c9b5062bdfa46af9396f1a85025a331. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d416e1015a2644958b0c8f9147d403a2. This is attempt 1: this message will repeat every 5th retry.
I20260807 03:39:53.089157 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:53.090960 18246 leader_election.cc:302] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:39:53.041385 22658 leader_election.cc:288] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:53.093480 18249 leader_election.cc:302] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:53.101775 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:53.103420 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:53.055213 22530 leader_election.cc:288] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:53.110831 19297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:53.064834 22975 raft_consensus.cc:2781] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 03:39:53.086979 23017 raft_consensus.cc:482] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:53.113957 18249 leader_election.cc:302] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:53.088860 22931 raft_consensus.cc:2781] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 03:39:53.082852 22520 raft_consensus.cc:686] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:53.119462 18297 raft_consensus.cc:2379] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 10: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:53.128188 19297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:53.088160 22776 raft_consensus.cc:482] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5c9b5062bdfa46af9396f1a85025a331)
I20260807 03:39:53.104199 23046 raft_consensus.cc:2781] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:39:53.031181 22865 raft_consensus.cc:504] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:53.103782 22987 raft_consensus.cc:2726] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260807 03:39:53.147212 18312 raft_consensus.cc:1261] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260807 03:39:53.147794 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:53.148401 18191 raft_consensus.cc:2454] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 8.
I20260807 03:39:53.143015 22532 raft_consensus.cc:2781] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:39:53.151693 18249 leader_election.cc:302] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:53.041174 22888 consensus_queue.cc:260] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:53.114145 22975 raft_consensus.cc:482] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:53.153643 19297 raft_consensus.cc:1261] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331 [term 4 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 03:39:53.113418 22886 raft_consensus.cc:482] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:53.153262 22532 raft_consensus.cc:482] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:53.144941 23046 raft_consensus.cc:482] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:53.149152 22996 raft_consensus.cc:482] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:53.152328 23042 raft_consensus.cc:2781] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election won for term 5
W20260807 03:39:53.165273 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 191916us
I20260807 03:39:53.156354 23041 raft_consensus.cc:2726] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:53.139997 23017 raft_consensus.cc:504] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:53.160327 22781 raft_consensus.cc:2781] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 03:39:53.159782 22975 raft_consensus.cc:3037] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:53.176674 18305 raft_consensus.cc:2454] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 8.
I20260807 03:39:53.144902 22776 raft_consensus.cc:504] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:53.164188 22532 raft_consensus.cc:3037] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:53.179208 18310 raft_consensus.cc:1261] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260807 03:39:53.216014 18297 raft_consensus.cc:1261] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Refusing update from remote peer 5c9b5062bdfa46af9396f1a85025a331: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260807 03:39:53.180471 18305 raft_consensus.cc:1226] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:53.196506 22980 consensus_queue.cc:1035] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 03:39:53.199254 18191 raft_consensus.cc:1226] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 7. Current term is 8. Ops: []
I20260807 03:39:53.200340 18293 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37190 (request call id 176) took 1230 ms. Trace:
I20260807 03:39:53.138751 22853 consensus_queue.cc:237] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:53.166004 23046 raft_consensus.cc:3037] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:53.163982 22886 raft_consensus.cc:504] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:53.209933 18311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37190 (request call id 173) took 1439 ms. Trace:
I20260807 03:39:53.250099 18311 rpcz_store.cc:276] 0807 03:39:51.770472 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:51.771191 (+ 719us) service_pool.cc:227] Handling call
0807 03:39:53.209679 (+1438488us) spinlock_profiling.cc:243] Waited 29.1 ms on lock 0x7b54002b48d4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:53.209845 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":66889472}
I20260807 03:39:53.202276 18072 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33254 (request call id 2337) took 1403 ms. Trace:
I20260807 03:39:53.222077 18293 rpcz_store.cc:276] 0807 03:39:51.969523 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:51.969678 (+ 155us) service_pool.cc:227] Handling call
0807 03:39:53.200119 (+1230441us) spinlock_profiling.cc:243] Waited 187 ms on lock 0x7b54001c8c54. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:53.200240 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":430042496}
I20260807 03:39:53.232936 18140 leader_election.cc:302] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:53.234931 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:53.238561 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:53.232903 18027 leader_election.cc:302] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:53.253543 18027 leader_election.cc:302] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:53.251113 18072 rpcz_store.cc:276] 0807 03:39:51.798752 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:51.798887 (+ 135us) service_pool.cc:227] Handling call
0807 03:39:53.202074 (+1403187us) spinlock_profiling.cc:243] Waited 614 ms on lock 0x7b54001653d4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:53.202187 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1411484928}
I20260807 03:39:53.208762 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:53.256475 18071 raft_consensus.cc:2454] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 7.
I20260807 03:39:53.249459 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:53.257822 18080 raft_consensus.cc:2454] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 4.
I20260807 03:39:53.174239 22781 raft_consensus.cc:482] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:53.170105 22996 raft_consensus.cc:504] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:53.217603 22658 raft_consensus.cc:482] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:53.261929 18140 leader_election.cc:302] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:53.266827 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:53.244957 22902 raft_consensus.cc:3032] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Stepping down as leader of term 5
I20260807 03:39:53.282189 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:53.273828 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:53.276154 23016 raft_consensus.cc:482] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:53.246637 22882 raft_consensus.cc:482] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:53.289008 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:53.285085 18071 raft_consensus.cc:2454] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 6.
I20260807 03:39:53.274307 18079 raft_consensus.cc:2454] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 7.
I20260807 03:39:53.291792 18075 raft_consensus.cc:2379] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 5: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:53.297369 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:53.297426 18140 leader_election.cc:302] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:53.298237 18075 raft_consensus.cc:2379] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 5: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:53.300319 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:53.301083 18187 raft_consensus.cc:2454] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 9.
I20260807 03:39:53.303282 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:53.304534 18312 raft_consensus.cc:1226] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 9. Current term is 10. Ops: []
I20260807 03:39:53.306355 18185 raft_consensus.cc:1261] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20260807 03:39:53.306420 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:53.306833 18185 pending_rounds.cc:76] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02: Aborting all ops after (but not including) 4
I20260807 03:39:53.307112 18185 pending_rounds.cc:97] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260807 03:39:53.307082 18187 raft_consensus.cc:2454] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 5.
I20260807 03:39:53.307493 18185 raft_consensus.cc:2866] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 03:39:53.261734 22988 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:37176 (request call id 6) took 2359 ms. Trace:
I20260807 03:39:53.317981 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:53.267964 22932 raft_consensus.cc:2726] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260807 03:39:53.260001 22781 raft_consensus.cc:3037] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:53.319711 18191 raft_consensus.cc:1226] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 7. Current term is 8. Ops: []
I20260807 03:39:53.250556 22993 consensus_queue.cc:1046] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.042s
I20260807 03:39:53.271708 22857 raft_consensus.cc:2781] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader election won for term 8
I20260807 03:39:53.255781 22987 consensus_queue.cc:1046] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.048s
I20260807 03:39:53.270496 22441 leader_election.cc:288] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:53.287595 22902 raft_consensus.cc:729] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:53.329433 18183 raft_consensus.cc:1261] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260807 03:39:53.288084 22921 raft_consensus.cc:2781] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 03:39:53.315990 22908 raft_consensus.cc:2781] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 03:39:53.296205 22658 raft_consensus.cc:504] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:53.316341 22882 raft_consensus.cc:504] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:53.344026 23017 leader_election.cc:288] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:53.344602 23025 leader_election.cc:288] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:53.321201 22988 rpcz_store.cc:276] 0807 03:39:50.895668 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:50.895794 (+ 126us) service_pool.cc:227] Handling call
0807 03:39:51.034113 (+138319us) log.cc:842] Serialized 513 byte log entry
0807 03:39:51.245411 (+211298us) spinlock_profiling.cc:243] Waited 2.89 ms on lock 0x7b18000f98e0. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe9981368b1 00007fe998143661 00007fe98f20e4ca 00007fe9988eff99 00007fe98f0d2990 00007fe98f294864 00007fe98f294155 00007fe992139320 00007fe99219f113 00007fe9921c57e2 00007fe997101f39
0807 03:39:51.419128 (+173717us) spinlock_profiling.cc:243] Waited 12.6 ms on lock 0x7b18000f98e0. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe9981368b1 00007fe998143661 00007fe98f264f96 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe9981368b1 00007fe998143661 00007fe98f297638 00007fe98f29ab11
0807 03:39:51.597388 (+178260us) spinlock_profiling.cc:243] Waited 2.27 ms on lock 0x7b18000f98e0. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe9981368b1 00007fe998143661 00007fe98f297638 00007fe98f29ab11 00007fe98f29aac9 00007fe98f29aa59 00007fe98f29aa21 00007fe98f299d1d 00007fe9984fb204 00007fe9984fb039 00007fe98f279dc6
0807 03:39:51.672641 (+ 75253us) spinlock_profiling.cc:243] Waited 1.06 ms on lock 0x7b2c00108448. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe990ba2e11 00007fe990bb1ada 00007fe990bb1763 00007fe990bb1701 00007fe9921fbd62 00007fe9921fbf05 00007fe992150f91 00007fe992139545 00007fe99213e771 00007fe99213e6f9 00007fe99213e689
0807 03:39:51.673320 (+ 679us) spinlock_profiling.cc:243] Waited 6.41 ms on lock 0x7b18000b2860. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921fbf11 00007fe992150f91 00007fe992139545 00007fe99213e771 00007fe99213e6f9 00007fe99213e689 00007fe99213e651 00007fe99213d81d 00007fe9984fb204 00007fe9984fb039
0807 03:39:52.319002 (+645682us) spinlock_profiling.cc:243] Waited 408 ms on lock 0x7b18000b2860. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921fbf11 00007fe992150f91 00007fe992139545 00007fe99213e771 00007fe99213e6f9 00007fe99213e689 00007fe99213e651 00007fe99213d81d 00007fe9984fb204 00007fe9984fb039
0807 03:39:53.250892 (+931890us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":66735,"raft.queue_time_us":69577,"raft.run_cpu_time_us":14275,"raft.run_wall_time_us":1527026,"spinlock_wait_cycles":995447552,"thread_start_us":159140,"threads_started":1,"wal-append.queue_time_us":215533,"wal-append.run_cpu_time_us":1526,"wal-append.run_wall_time_us":1169772}
W20260807 03:39:53.347581 23020 log.cc:924] Time spent T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 1.011s user 0.000s sys 0.010s
I20260807 03:39:53.349398 22996 leader_election.cc:288] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:53.217796 22520 consensus_queue.cc:237] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:53.354808 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:53.355196 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:53.271253 23055 raft_consensus.cc:2781] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260807 03:39:53.359009 18246 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.132:41381, user_credentials={real_user=slave}} blocked reactor thread for 186263us
W20260807 03:39:53.360185 19239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 134619us
I20260807 03:39:53.286484 23016 raft_consensus.cc:504] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:53.362711 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:53.458079 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:53.458463 18079 raft_consensus.cc:2354] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 3. Current term is 4.
I20260807 03:39:53.370803 18075 raft_consensus.cc:1261] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260807 03:39:53.329479 23001 raft_consensus.cc:482] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5c9b5062bdfa46af9396f1a85025a331)
I20260807 03:39:53.344625 22865 leader_election.cc:288] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:53.419781 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:53.453845 19239 leader_election.cc:302] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
W20260807 03:39:53.455279 19240 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 275478us
I20260807 03:39:53.392407 22908 raft_consensus.cc:482] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:53.374707 22921 raft_consensus.cc:482] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:53.365285 22857 raft_consensus.cc:686] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 8 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:53.460453 18305 raft_consensus.cc:1261] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260807 03:39:53.460554 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:53.398492 22934 raft_consensus.cc:482] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:53.560194 22934 raft_consensus.cc:504] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:53.487833 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:53.561750 18075 raft_consensus.cc:2379] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 9: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:53.553172 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:53.572399 22776 leader_election.cc:288] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:53.456734 22866 raft_consensus.cc:504] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:53.576275 18309 raft_consensus.cc:2454] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 5.
I20260807 03:39:53.456926 23055 raft_consensus.cc:482] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:53.579476 23055 raft_consensus.cc:3037] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:53.585402 22658 leader_election.cc:288] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:53.587589 23054 consensus_queue.cc:1035] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260807 03:39:53.588587 19297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:53.556005 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:53.575755 22866 leader_election.cc:288] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
W20260807 03:39:53.607239 19239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 153499us
I20260807 03:39:53.461592 22996 raft_consensus.cc:482] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5c9b5062bdfa46af9396f1a85025a331)
I20260807 03:39:53.684194 19239 leader_election.cc:302] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:53.699129 18075 raft_consensus.cc:1261] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 03:39:53.554692 22908 raft_consensus.cc:3037] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:53.555022 23080 raft_consensus.cc:482] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:53.554885 22921 raft_consensus.cc:3037] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:53.554284 22865 raft_consensus.cc:482] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:53.687526 22776 raft_consensus.cc:482] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:53.720834 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:53.672024 22886 leader_election.cc:288] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:53.726524 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:53.487164 23001 raft_consensus.cc:504] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
W20260807 03:39:53.716205 23070 log.cc:924] Time spent T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.368s user 0.000s sys 0.001s
I20260807 03:39:53.744208 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:53.412493 23042 raft_consensus.cc:686] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:53.485710 22902 consensus_queue.cc:260] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:53.791976 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 304079us
W20260807 03:39:53.794059 19240 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 303687us
I20260807 03:39:53.745023 18072 raft_consensus.cc:2454] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:53.761938 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:53.711310 23089 raft_consensus.cc:2781] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 03:39:53.786458 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:53.796968 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:53.788794 19239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 104694us
I20260807 03:39:53.702642 22996 raft_consensus.cc:504] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:53.740103 23059 raft_consensus.cc:482] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:53.830443 18072 raft_consensus.cc:1226] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 6. Current term is 7. Ops: []
I20260807 03:39:53.733256 22776 raft_consensus.cc:504] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:53.727402 22865 raft_consensus.cc:504] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:53.798985 23089 raft_consensus.cc:482] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 03:39:53.849282 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 56504.8us
I20260807 03:39:53.723520 23080 raft_consensus.cc:504] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:53.815779 22990 raft_consensus.cc:482] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:53.827203 22902 raft_consensus.cc:3037] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:53.831195 22658 raft_consensus.cc:482] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:53.843060 23025 raft_consensus.cc:482] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:53.879058 18079 raft_consensus.cc:2379] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 5: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:53.881697 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:53.882754 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ca0223f47b46fd955704eba5259370" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:53.841236 22441 raft_consensus.cc:482] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:53.891712 18311 raft_consensus.cc:2379] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 9: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:53.790261 22857 consensus_queue.cc:237] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:53.838480 23059 raft_consensus.cc:504] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:53.879130 22739 consensus_meta.cc:287] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 0.787s user 0.006s sys 0.007s
I20260807 03:39:53.852342 23089 raft_consensus.cc:3037] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331 [term 7 FOLLOWER]: Advancing to term 8
W20260807 03:39:53.937209 19240 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 142422us
I20260807 03:39:53.826902 23101 consensus_queue.cc:1035] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.122s
W20260807 03:39:53.946993 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 94963.1us
I20260807 03:39:53.950881 22132 raft_consensus.cc:482] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:53.951467 22132 raft_consensus.cc:504] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:53.912846 22986 raft_consensus.cc:482] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:53.954797 18027 leader_election.cc:302] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:53.954195 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ca0223f47b46fd955704eba5259370" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:53.952574 18080 raft_consensus.cc:2454] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 5.
I20260807 03:39:53.893612 22882 leader_election.cc:288] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:53.925086 23092 raft_consensus.cc:3032] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Stepping down as leader of term 7
I20260807 03:39:53.965198 23092 raft_consensus.cc:729] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:53.966406 23092 consensus_queue.cc:260] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:53.967962 23092 raft_consensus.cc:3037] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:53.975560 18027 leader_election.cc:302] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:53.982764 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:53.890897 22658 raft_consensus.cc:504] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:53.908106 23025 raft_consensus.cc:504] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:53.988308 18075 raft_consensus.cc:2379] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 5: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:53.952417 23116 raft_consensus.cc:482] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:53.912653 22441 raft_consensus.cc:504] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:39:53.995743 18248 leader_election.cc:302] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:53.886358 22990 raft_consensus.cc:504] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:39:54.000200 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:54.002952 23098 log.cc:924] Time spent T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.090s user 0.000s sys 0.001s
I20260807 03:39:54.003068 23041 raft_consensus.cc:482] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 03:39:54.003369 23088 log.cc:924] Time spent T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.209s user 0.001s sys 0.000s
I20260807 03:39:53.979092 23016 leader_election.cc:288] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:54.007047 23078 log.cc:924] Time spent T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.278s user 0.000s sys 0.001s
I20260807 03:39:54.009282 18248 leader_election.cc:398] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Vote denied by peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163) with higher term. Message: Invalid argument: T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 3. Current term is 4.
I20260807 03:39:54.009495 18183 raft_consensus.cc:1226] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 7. Current term is 8. Ops: []
I20260807 03:39:54.005168 18310 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57268 (request call id 2100) took 1029 ms. Trace:
I20260807 03:39:54.012354 18248 leader_election.cc:401] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260807 03:39:54.012957 18310 rpcz_store.cc:276] 0807 03:39:52.975547 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:52.975710 (+ 163us) service_pool.cc:227] Handling call
0807 03:39:53.178888 (+203178us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:53.215344 (+ 36456us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:54.004952 (+789608us) spinlock_profiling.cc:243] Waited 203 ms on lock 0x7b540018bfd0. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bea09 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:54.005089 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":466402560}
I20260807 03:39:54.026036 18248 leader_election.cc:302] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:54.004177 22865 leader_election.cc:288] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:53.974354 22986 raft_consensus.cc:504] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:54.004925 23001 leader_election.cc:288] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:53.988921 23122 raft_consensus.cc:2781] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:54.048434 18180 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44660 (request call id 2499) took 2202 ms. Trace:
I20260807 03:39:54.049062 18180 rpcz_store.cc:276] 0807 03:39:51.840269 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:51.840430 (+ 161us) service_pool.cc:227] Handling call
0807 03:39:54.041084 (+2200654us) spinlock_profiling.cc:243] Waited 1.13 ms on lock 0x7b5800003998. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f25f851 00007fe98f25fbce 000000000034aea9 00007fe99708ba30 00007fe9971517b4 00007fe997151821 00007fe9971152b0 00007fe9971007f9 00007fe9971002a2 00007fe9907e6684
0807 03:39:54.041906 (+ 822us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:54.042078 (+ 172us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:54.042101 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:54.042151 (+ 50us) raft_consensus.cc:1631] Marking committed up to 3
0807 03:39:54.042187 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:54.042232 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:54.042317 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2602368}
I20260807 03:39:54.012233 23045 raft_consensus.cc:2726] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260807 03:39:54.055205 23085 log.cc:924] Time spent T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.255s user 0.001s sys 0.001s
I20260807 03:39:54.057473 23109 consensus_queue.cc:1035] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20260807 03:39:54.053191 22996 leader_election.cc:288] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:54.058888 22996 raft_consensus.cc:2726] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260807 03:39:54.060621 23013 raft_consensus.cc:3032] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 7 LEADER]: Stepping down as leader of term 7
I20260807 03:39:54.061043 23013 raft_consensus.cc:729] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:54.061853 23013 consensus_queue.cc:260] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:54.062884 23013 raft_consensus.cc:3037] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Advancing to term 8
W20260807 03:39:54.064788 23094 log.cc:924] Time spent T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.229s user 0.000s sys 0.001s
I20260807 03:39:54.065281 23103 raft_consensus.cc:2726] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163) with higher term. Message: Invalid argument: T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 3. Current term is 4.
I20260807 03:39:54.048923 22865 raft_consensus.cc:2781] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 03:39:53.948812 23042 consensus_queue.cc:237] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:54.052316 23122 raft_consensus.cc:482] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:54.053529 22776 leader_election.cc:288] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:54.066499 23101 raft_consensus.cc:2781] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260807 03:39:54.116570 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 138275us
I20260807 03:39:54.114547 18069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33254 (request call id 2320) took 2691 ms. Trace:
I20260807 03:39:54.071862 22658 leader_election.cc:288] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:54.115464 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 89417.1us
I20260807 03:39:54.099758 22865 raft_consensus.cc:482] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:54.013778 23041 raft_consensus.cc:504] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:54.005218 23116 raft_consensus.cc:504] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:54.121722 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 198884us
W20260807 03:39:54.136422 22781 consensus_meta.cc:287] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.775s user 0.012s sys 0.000s
I20260807 03:39:54.253798 18248 leader_election.cc:302] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:54.118439 18069 rpcz_store.cc:276] 0807 03:39:51.418682 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:51.418826 (+ 144us) service_pool.cc:227] Handling call
0807 03:39:51.419118 (+ 292us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:51.424850 (+ 5732us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:51.424880 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:39:51.436301 (+ 11421us) log.cc:842] Serialized 26 byte log entry
0807 03:39:53.205587 (+1769286us) spinlock_profiling.cc:243] Waited 24.4 ms on lock 0x7b1800057040. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe9981368b1 00007fe998143661 00007fe98f279345 00007fe9980b0a39 00007fe98f2968bb 00007fe98f294cbf 00007fe98f296e22 00007fe991e405bb 00007fe991e46063 00007fe991e46411 00007fe9921935cf
0807 03:39:53.365268 (+159681us) raft_consensus.cc:1631] Marking committed up to 5
0807 03:39:53.365318 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:53.365425 (+ 107us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:54.100256 (+734831us) raft_consensus.cc:1671] finished
0807 03:39:54.100277 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:54.110042 (+ 9765us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":6458,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":88,"spinlock_wait_cycles":56206080,"thread_start_us":159463,"threads_started":1,"wal-append.queue_time_us":2280326}
I20260807 03:39:54.247013 22986 leader_election.cc:288] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:54.256155 23083 consensus_queue.cc:1035] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.211s
I20260807 03:39:54.257115 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:54.251940 23115 raft_consensus.cc:482] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:54.258139 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:54.259485 18069 raft_consensus.cc:2379] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 7: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:54.260077 18140 leader_election.cc:302] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:54.257997 23115 raft_consensus.cc:504] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:54.258410 22990 leader_election.cc:288] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:54.254822 22781 raft_consensus.cc:504] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:54.252355 23130 consensus_queue.cc:1046] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.094s
W20260807 03:39:54.255514 22921 consensus_meta.cc:287] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 0.527s user 0.007s sys 0.003s
W20260807 03:39:54.251037 23046 consensus_meta.cc:287] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.991s user 0.004s sys 0.004s
I20260807 03:39:54.122447 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:39:54.209388 18003 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica 3c4bceb836db4536a0f255fdf86abc02 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:54.261776 19297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:54.296432 18180 raft_consensus.cc:2454] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 9.
I20260807 03:39:54.270282 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:54.297818 18249 leader_election.cc:302] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:54.298086 18069 raft_consensus.cc:2454] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 5.
I20260807 03:39:54.290518 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:39:54.294628 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:54.263974 18140 leader_election.cc:302] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:54.299953 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:54.304526 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 50810.2us
I20260807 03:39:54.108781 23001 raft_consensus.cc:482] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 03:39:54.226136 23090 log.cc:924] Time spent T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.223s user 0.000s sys 0.001s
W20260807 03:39:54.227712 23093 log.cc:924] Time spent T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.108s user 0.000s sys 0.000s
I20260807 03:39:54.305680 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:54.109345 23122 raft_consensus.cc:3037] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:54.322388 18310 raft_consensus.cc:1226] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 7. Current term is 9. Ops: []
I20260807 03:39:54.322719 19295 raft_consensus.cc:1261] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331 [term 8 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260807 03:39:54.323454 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:54.178850 22532 consensus_meta.cc:287] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.952s user 0.010s sys 0.002s
I20260807 03:39:54.337304 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:54.252728 23125 raft_consensus.cc:482] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:54.289762 22658 raft_consensus.cc:2781] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:39:54.327106 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:54.198804 22975 consensus_meta.cc:287] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.984s user 0.006s sys 0.006s
I20260807 03:39:54.332913 22427 consensus_queue.cc:1046] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 03:39:54.333377 19291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:54.336302 23104 raft_consensus.cc:482] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:54.262099 22996 raft_consensus.cc:482] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:54.337821 22532 raft_consensus.cc:504] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:54.348744 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:54.141999 22871 consensus_queue.cc:237] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 7, Current term: 9, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:54.315999 22986 raft_consensus.cc:2726] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 03:39:54.292048 22441 leader_election.cc:288] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:54.117497 23101 raft_consensus.cc:482] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:54.295295 22921 raft_consensus.cc:504] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:54.301288 23025 leader_election.cc:288] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:54.362012 18070 raft_consensus.cc:1261] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Refusing update from remote peer 5c9b5062bdfa46af9396f1a85025a331: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 03:39:54.322145 23045 raft_consensus.cc:2781] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260807 03:39:54.363565 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:54.331334 23001 raft_consensus.cc:504] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:54.120632 23080 leader_election.cc:288] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:54.209633 23059 leader_election.cc:288] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:54.321839 23055 consensus_meta.cc:287] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 0.741s user 0.009s sys 0.004s
I20260807 03:39:54.380142 23127 raft_consensus.cc:482] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5c9b5062bdfa46af9396f1a85025a331)
I20260807 03:39:54.381593 18073 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33268 (request call id 2021) took 1879 ms. Trace:
I20260807 03:39:54.383517 22532 leader_election.cc:288] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:54.379562 19290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:54.362890 23125 raft_consensus.cc:504] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:54.384878 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:54.365842 23013 raft_consensus.cc:482] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:54.226312 23124 consensus_queue.cc:1035] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.288s
I20260807 03:39:54.386466 23116 leader_election.cc:288] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:54.367726 22909 raft_consensus.cc:482] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:54.217564 23137 consensus_queue.cc:1035] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.161s
I20260807 03:39:54.447150 23001 leader_election.cc:288] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:54.447034 22909 raft_consensus.cc:504] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:54.387501 23134 raft_consensus.cc:2781] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 03:39:54.388401 23126 consensus_queue.cc:1035] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260807 03:39:54.389925 22781 leader_election.cc:288] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:54.390272 23115 leader_election.cc:288] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:54.392094 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 92094.7us
W20260807 03:39:54.425542 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 108744us
I20260807 03:39:54.489866 22132 leader_election.cc:288] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:54.351121 23130 raft_consensus.cc:482] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5c9b5062bdfa46af9396f1a85025a331)
W20260807 03:39:54.397670 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 72086.2us
I20260807 03:39:54.398280 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:54.363219 22990 raft_consensus.cc:482] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 03:39:54.357528 19240 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 410040us
I20260807 03:39:54.358670 22658 raft_consensus.cc:482] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:54.324402 22865 raft_consensus.cc:3037] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:54.409970 23101 raft_consensus.cc:3037] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:54.303516 23046 raft_consensus.cc:504] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:54.438441 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:54.366163 22975 raft_consensus.cc:504] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:54.430121 18073 rpcz_store.cc:276] 0807 03:39:52.501938 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:52.502464 (+ 526us) service_pool.cc:227] Handling call
0807 03:39:52.502876 (+ 412us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 03:39:52.524446 (+ 21570us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:52.524489 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 03:39:52.525157 (+ 668us) log.cc:842] Serialized 52 byte log entry
0807 03:39:53.244604 (+719447us) spinlock_profiling.cc:243] Waited 7.62 ms on lock 0x7b18003473e0. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe9981368b1 00007fe998143661 00007fe98f279345 00007fe9980b0a39 00007fe98f2968bb 00007fe98f294cbf 00007fe98f296e22 00007fe991e405bb 00007fe991e46063 00007fe991e46411 00007fe9921935cf
0807 03:39:53.424409 (+179805us) raft_consensus.cc:1631] Marking committed up to 4
0807 03:39:53.424455 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:53.424551 (+ 96us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:54.381275 (+956724us) raft_consensus.cc:1671] finished
0807 03:39:54.381296 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:54.381533 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":21196,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":17657,"spinlock_wait_cycles":17532288,"thread_start_us":179559,"threads_started":1,"wal-append.queue_time_us":1494406}
I20260807 03:39:54.430264 19291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:54.427171 23089 raft_consensus.cc:504] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:54.428207 23136 raft_consensus.cc:482] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:54.426930 23041 leader_election.cc:288] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:54.390834 22996 raft_consensus.cc:504] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:54.527563 18305 raft_consensus.cc:2454] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 9.
I20260807 03:39:54.414258 23045 raft_consensus.cc:482] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:54.445928 23125 leader_election.cc:288] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:54.446499 22921 leader_election.cc:288] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:54.448323 23134 raft_consensus.cc:482] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:54.524212 23092 raft_consensus.cc:482] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:54.501175 18183 raft_consensus.cc:1226] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 7. Current term is 8. Ops: []
I20260807 03:39:54.525741 23001 raft_consensus.cc:482] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:54.514600 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:54.391052 23104 raft_consensus.cc:504] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:54.502923 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:54.582307 18073 raft_consensus.cc:3037] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:54.582450 18187 raft_consensus.cc:2454] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 5.
I20260807 03:39:54.515895 18248 leader_election.cc:302] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:54.500449 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:54.492295 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:54.429535 23055 raft_consensus.cc:504] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:54.542956 18293 raft_consensus.cc:2379] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 6: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
W20260807 03:39:54.520709 22908 consensus_meta.cc:287] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 0.796s user 0.006s sys 0.006s
I20260807 03:39:54.543746 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:54.515894 22658 raft_consensus.cc:3037] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:54.490743 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:54.629429 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:54.630064 18183 raft_consensus.cc:2454] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 9.
I20260807 03:39:54.503814 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:54.573194 22909 leader_election.cc:288] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:54.572928 22917 consensus_queue.cc:1035] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.122s
I20260807 03:39:54.622974 18073 raft_consensus.cc:2454] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 9.
I20260807 03:39:54.574452 22762 raft_consensus.cc:482] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:54.574788 22781 raft_consensus.cc:482] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:54.566856 23134 raft_consensus.cc:3037] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:54.577329 18312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57276 (request call id 1733) took 1428 ms. Trace:
I20260807 03:39:54.584220 18180 raft_consensus.cc:2454] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 9.
I20260807 03:39:54.544932 23045 raft_consensus.cc:3037] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Advancing to term 10
I20260807 03:39:54.443518 23013 raft_consensus.cc:504] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:54.648066 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:54.649586 18187 raft_consensus.cc:2379] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 8: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:54.652021 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:54.646417 18312 rpcz_store.cc:276] 0807 03:39:53.148387 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:53.213003 (+ 64616us) service_pool.cc:227] Handling call
0807 03:39:53.230577 (+ 17574us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:53.305036 (+ 74459us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:54.577080 (+1272044us) spinlock_profiling.cc:243] Waited 73.7 ms on lock 0x7b54002634d4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:39:54.577239 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":169444096}
I20260807 03:39:54.652667 18188 raft_consensus.cc:2454] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 4.
I20260807 03:39:54.655673 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:54.656390 18187 raft_consensus.cc:3037] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:54.659142 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:54.659935 18312 raft_consensus.cc:2379] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 6: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:54.660794 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:54.661381 18312 raft_consensus.cc:2454] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 4.
I20260807 03:39:54.662132 23003 leader_election.cc:288] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:54.662132 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:54.674953 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:54.581053 23092 raft_consensus.cc:504] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:54.633244 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 117441us
W20260807 03:39:54.681722 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 115446us
I20260807 03:39:54.682644 18248 leader_election.cc:302] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:54.429834 23127 raft_consensus.cc:504] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:54.635661 22975 leader_election.cc:288] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:54.635778 23149 raft_consensus.cc:2726] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:54.523489 22990 raft_consensus.cc:504] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:54.627789 18310 raft_consensus.cc:2454] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 8.
I20260807 03:39:54.641579 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:54.640460 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:54.645677 18191 raft_consensus.cc:2354] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 5. Current term is 6.
I20260807 03:39:54.640712 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:54.646947 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:54.631783 18185 raft_consensus.cc:2454] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 7.
I20260807 03:39:54.699095 18180 raft_consensus.cc:3037] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:54.524968 23130 raft_consensus.cc:504] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:54.655403 22817 raft_consensus.cc:1013] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 9 LEADER]: attempt to promote peer 5c9b5062bdfa46af9396f1a85025a331: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260807 03:39:54.616669 23001 raft_consensus.cc:504] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:54.540897 23136 raft_consensus.cc:504] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:54.652061 23104 leader_election.cc:288] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:54.684733 18187 raft_consensus.cc:2454] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 6.
I20260807 03:39:54.631062 22908 raft_consensus.cc:504] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:54.696302 18027 leader_election.cc:302] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:39:54.670643 23126 raft_consensus.cc:482] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:54.648483 22762 raft_consensus.cc:504] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:54.680266 23042 raft_consensus.cc:482] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:54.718067 18295 raft_consensus.cc:1226] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 7. Current term is 8. Ops: []
I20260807 03:39:54.720187 18140 leader_election.cc:302] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:54.655552 22781 raft_consensus.cc:504] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:54.762259 23138 consensus_queue.cc:1035] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 03:39:54.722079 18069 raft_consensus.cc:2454] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 11.
I20260807 03:39:54.723229 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:54.726835 23055 leader_election.cc:288] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:54.728039 23046 leader_election.cc:288] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
W20260807 03:39:54.728729 23023 log.cc:924] Time spent T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 2.505s user 0.015s sys 0.015s
I20260807 03:39:54.730400 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:54.700430 23041 consensus_queue.cc:1046] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.049s
I20260807 03:39:54.732925 23013 leader_election.cc:288] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:54.781495 22841 raft_consensus.cc:482] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:54.782039 22841 raft_consensus.cc:504] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:54.783576 22916 raft_consensus.cc:482] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 5c9b5062bdfa46af9396f1a85025a331)
I20260807 03:39:54.784674 23161 raft_consensus.cc:482] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 5c9b5062bdfa46af9396f1a85025a331)
W20260807 03:39:54.742236 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 46002.9us
I20260807 03:39:54.786623 18027 leader_election.cc:302] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:54.743153 23116 raft_consensus.cc:2781] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:54.744585 18193 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44662 (request call id 1680) took 2255 ms. Trace:
I20260807 03:39:54.787333 18193 rpcz_store.cc:276] 0807 03:39:52.489108 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:52.501026 (+ 11918us) service_pool.cc:227] Handling call
0807 03:39:52.501304 (+ 278us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:53.058085 (+556781us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:54.744380 (+1686295us) spinlock_profiling.cc:243] Waited 519 ms on lock 0x7b5400172d54. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:39:54.744530 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1193817984}
I20260807 03:39:54.787756 18310 raft_consensus.cc:1261] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Refusing update from remote peer 5c9b5062bdfa46af9396f1a85025a331: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 03:39:54.744894 22908 leader_election.cc:288] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:54.748354 18073 raft_consensus.cc:2454] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 9.
I20260807 03:39:54.723826 23149 raft_consensus.cc:2726] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260807 03:39:54.761078 23155 raft_consensus.cc:482] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5c9b5062bdfa46af9396f1a85025a331)
I20260807 03:39:54.758531 23042 raft_consensus.cc:504] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:54.761757 23015 raft_consensus.cc:482] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:54.765339 18183 raft_consensus.cc:2379] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 6: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:54.774185 18070 raft_consensus.cc:3037] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:54.731312 22980 raft_consensus.cc:482] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:54.762835 23127 leader_election.cc:288] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:54.748067 23124 raft_consensus.cc:482] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:54.812940 19292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
W20260807 03:39:54.820345 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 119994us
I20260807 03:39:54.745983 23134 raft_consensus.cc:504] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:54.792483 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 75420.3us
I20260807 03:39:54.811826 23153 ts_tablet_manager.cc:927] T d416e1015a2644958b0c8f9147d403a2 P 5c9b5062bdfa46af9396f1a85025a331: Initiating tablet copy from peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:54.764828 23089 leader_election.cc:288] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 8 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:54.778956 22781 leader_election.cc:288] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:54.785228 23161 raft_consensus.cc:504] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
W20260807 03:39:54.822705 18141 connection.cc:232] RPC call timeout handler was delayed by -1.5527731890324503s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1644s for a response
I20260807 03:39:54.804998 23017 raft_consensus.cc:482] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:54.855587 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:54.874212 19240 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 48057.6us
I20260807 03:39:54.853060 23158 raft_consensus.cc:482] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 5c9b5062bdfa46af9396f1a85025a331)
I20260807 03:39:54.807861 22921 raft_consensus.cc:2781] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:39:54.820715 23001 leader_election.cc:288] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:54.805184 23116 raft_consensus.cc:482] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:54.777007 22996 leader_election.cc:288] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
W20260807 03:39:54.872450 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 43739.9us
I20260807 03:39:54.813277 23013 raft_consensus.cc:2781] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 03:39:54.852622 22841 leader_election.cc:288] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
W20260807 03:39:54.873576 18141 connection.cc:232] RPC call timeout handler was delayed by -1.3849080491345376s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19440000000000002s for a response
I20260807 03:39:54.875474 18310 raft_consensus.cc:2454] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 6.
I20260807 03:39:54.820883 22857 raft_consensus.cc:482] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:54.881574 18070 raft_consensus.cc:2454] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 8.
I20260807 03:39:54.759028 23126 raft_consensus.cc:504] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:54.907330 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:54.853932 23127 raft_consensus.cc:482] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:54.824180 22908 raft_consensus.cc:482] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:54.915004 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:54.812485 23155 raft_consensus.cc:504] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:54.894189 18249 leader_election.cc:302] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
W20260807 03:39:54.903627 18141 connection.cc:232] RPC call timeout handler was delayed by -1.3746847058646381s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1756s for a response
W20260807 03:39:54.916231 18141 connection.cc:232] RPC call timeout handler was delayed by -1.2688080491498113s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2073s for a response
W20260807 03:39:54.916599 18141 connection.cc:232] RPC call timeout handler was delayed by -1.2526080491952598s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20910000000000004s for a response
W20260807 03:39:54.916962 18141 connection.cc:232] RPC call timeout handler was delayed by -1.1293080491013825s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22280000000000003s for a response
I20260807 03:39:54.814563 22916 raft_consensus.cc:504] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:54.912496 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:54.824478 23042 leader_election.cc:288] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:54.886502 22921 raft_consensus.cc:482] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:54.917948 18141 leader_election.cc:302] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:54.895601 23041 raft_consensus.cc:482] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:54.828011 22980 raft_consensus.cc:504] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:54.921077 22902 consensus_meta.cc:287] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 1.042s user 0.004s sys 0.008s
I20260807 03:39:54.827106 23015 raft_consensus.cc:504] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:54.961093 22762 leader_election.cc:288] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
W20260807 03:39:54.962595 23146 log.cc:924] Time spent T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
I20260807 03:39:54.967710 18071 raft_consensus.cc:2454] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 6.
W20260807 03:39:54.940013 18248 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.15.198.129:36163 timed out after 1.875s (SENT) [suppressed 14 similar messages]
I20260807 03:39:54.919986 23092 leader_election.cc:288] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:54.922627 18310 raft_consensus.cc:2379] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 9: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:54.905757 18140 leader_election.cc:302] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:54.924969 19289 raft_consensus.cc:1261] T 6b66966247c147308a4b8488557e0bdf P 5c9b5062bdfa46af9396f1a85025a331 [term 9 LEARNER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260807 03:39:54.925084 23145 raft_consensus.cc:482] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:54.925992 23010 raft_consensus.cc:482] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 03:39:54.926648 18311 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57276 (request call id 1736) took 1468 ms (client timeout 1597 ms). Trace:
W20260807 03:39:54.989238 18311 rpcz_store.cc:269] 0807 03:39:53.458480 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:53.458600 (+ 120us) service_pool.cc:227] Handling call
0807 03:39:54.926412 (+1467812us) spinlock_profiling.cc:243] Waited 328 ms on lock 0x7b540035b454. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:54.926574 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":753885056}
I20260807 03:39:54.926901 22908 raft_consensus.cc:504] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:54.928921 18191 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44660 (request call id 2538) took 1049 ms. Trace:
I20260807 03:39:54.933988 23161 leader_election.cc:288] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:54.935376 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:54.936543 23126 leader_election.cc:288] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:54.937330 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:54.901829 23116 raft_consensus.cc:3037] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:54.944487 18141 leader_election.cc:302] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:54.903312 23013 raft_consensus.cc:482] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:54.938107 23149 raft_consensus.cc:2781] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 03:39:54.979591 23155 leader_election.cc:288] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:54.956518 23055 raft_consensus.cc:2726] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260807 03:39:54.950258 22921 raft_consensus.cc:3037] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:54.874140 23124 raft_consensus.cc:504] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:54.888975 23017 raft_consensus.cc:504] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:54.924386 22857 raft_consensus.cc:504] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:54.886343 23158 raft_consensus.cc:504] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:54.991602 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:55.002084 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:54.991964 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:54.997131 18072 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33254 (request call id 2371) took 1212 ms. Trace:
I20260807 03:39:55.116489 18075 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33254 (request call id 2373) took 1269 ms. Trace:
I20260807 03:39:55.137519 18081 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33254 (request call id 2372) took 1308 ms. Trace:
W20260807 03:39:55.137926 18300 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57268 (request call id 2096) took 2188 ms (client timeout 1756 ms). Trace:
I20260807 03:39:55.032035 18180 raft_consensus.cc:2454] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 9.
W20260807 03:39:55.165903 18080 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33268 (request call id 2034) took 1893 ms (client timeout 2135 ms). Trace:
I20260807 03:39:55.164857 18297 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37190 (request call id 177) took 2181 ms. Trace:
I20260807 03:39:55.175801 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:55.187204 18073 raft_consensus.cc:2454] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
I20260807 03:39:55.176703 18072 rpcz_store.cc:276] 0807 03:39:53.798238 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:53.840383 (+ 42145us) service_pool.cc:227] Handling call
0807 03:39:53.840700 (+ 317us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:53.887364 (+ 46664us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:53.887409 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:53.887464 (+ 55us) raft_consensus.cc:1631] Marking committed up to 5
0807 03:39:53.887507 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:54.996853 (+1109346us) spinlock_profiling.cc:243] Waited 46.4 ms on lock 0x7b5400033754. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:39:54.996910 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:54.997063 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":106755840}
I20260807 03:39:55.195472 22132 raft_consensus.cc:482] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:55.201910 19239 leader_election.cc:302] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
W20260807 03:39:55.182991 18300 rpcz_store.cc:269] 0807 03:39:52.949918 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:52.951962 (+ 2044us) service_pool.cc:227] Handling call
0807 03:39:55.137711 (+2185749us) spinlock_profiling.cc:243] Waited 13.3 ms on lock 0x7b54001d1d54. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:55.137827 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":30595968}
I20260807 03:39:55.173154 18189 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44662 (request call id 1687) took 1848 ms. Trace:
W20260807 03:39:55.184679 18080 rpcz_store.cc:269] 0807 03:39:53.272604 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:53.272819 (+ 215us) service_pool.cc:227] Handling call
0807 03:39:55.165719 (+1892900us) spinlock_profiling.cc:243] Waited 648 ms on lock 0x7b5400059454. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:55.165840 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1491435264}
I20260807 03:39:55.181660 18081 rpcz_store.cc:276] 0807 03:39:53.829392 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:53.845634 (+ 16242us) service_pool.cc:227] Handling call
0807 03:39:53.845950 (+ 316us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:54.032274 (+186324us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:54.032314 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:54.032358 (+ 44us) raft_consensus.cc:1631] Marking committed up to 8
0807 03:39:54.032399 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:55.137246 (+1104847us) spinlock_profiling.cc:243] Waited 186 ms on lock 0x7b54003d3bd4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:39:55.137301 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:55.137450 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":427878784}
I20260807 03:39:55.203151 18189 rpcz_store.cc:276] 0807 03:39:53.300187 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:53.301837 (+ 1650us) service_pool.cc:227] Handling call
0807 03:39:53.302163 (+ 326us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:53.680678 (+378515us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:53.680726 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:53.680779 (+ 53us) raft_consensus.cc:1631] Marking committed up to 4
0807 03:39:53.680833 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:55.148702 (+1467869us) spinlock_profiling.cc:243] Waited 378 ms on lock 0x7b54002a2e54. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:39:55.148758 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:55.148892 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":869871360}
I20260807 03:39:55.185665 18297 rpcz_store.cc:276] 0807 03:39:52.983783 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:53.128943 (+145160us) service_pool.cc:227] Handling call
0807 03:39:53.212396 (+ 83453us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:53.216492 (+ 4096us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:55.164620 (+1948128us) spinlock_profiling.cc:243] Waited 83.1 ms on lock 0x7b54002b48d0. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bea09 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:55.164787 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":191204096}
I20260807 03:39:55.103693 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:55.170713 18191 rpcz_store.cc:276] 0807 03:39:53.879816 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:53.881336 (+ 1520us) service_pool.cc:227] Handling call
0807 03:39:54.928680 (+1047344us) spinlock_profiling.cc:243] Waited 757 ms on lock 0x7b54001dea54. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:54.928839 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1741225344}
I20260807 03:39:55.207455 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:55.179629 18075 rpcz_store.cc:276] 0807 03:39:53.847564 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:54.013735 (+166171us) service_pool.cc:227] Handling call
0807 03:39:54.014023 (+ 288us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:54.302092 (+288069us) raft_consensus.cc:1520] Early marking committed up to index 5
0807 03:39:54.318505 (+ 16413us) log.cc:842] Serialized 14 byte log entry
0807 03:39:54.572323 (+253818us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:54.572392 (+ 69us) raft_consensus.cc:1631] Marking committed up to 5
0807 03:39:54.572458 (+ 66us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:55.116206 (+543748us) spinlock_profiling.cc:243] Waited 288 ms on lock 0x7b540005a854. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:39:55.116257 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:55.116408 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8229,"prepare.queue_time_us":16900,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":5027,"raft.queue_time_us":21484,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":84,"spinlock_wait_cycles":662013312,"thread_start_us":235989,"threads_started":1,"wal-append.queue_time_us":469943}
I20260807 03:39:55.218204 22132 raft_consensus.cc:504] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:55.219576 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:55.234867 18180 raft_consensus.cc:2454] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:39:55.236202 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:55.236966 18180 raft_consensus.cc:3037] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:55.221352 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:55.226040 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:55.238559 18072 raft_consensus.cc:2454] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 6.
I20260807 03:39:55.239630 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:55.227236 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:54.991436 23172 log.cc:924] Time spent T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.111s user 0.001s sys 0.000s
W20260807 03:39:55.061681 22865 consensus_meta.cc:287] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 0.536s user 0.008s sys 0.001s
W20260807 03:39:55.160414 23082 log.cc:924] Time spent T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 1.442s user 0.003s sys 0.002s
I20260807 03:39:55.176945 23013 raft_consensus.cc:3037] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Advancing to term 8
W20260807 03:39:55.111341 23181 log.cc:924] Time spent T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.123s user 0.000s sys 0.000s
I20260807 03:39:55.224203 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:54.965272 23041 raft_consensus.cc:504] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:55.251307 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:55.276209 18072 raft_consensus.cc:3037] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:54.946079 23127 raft_consensus.cc:504] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:55.252501 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:54.997946 22993 raft_consensus.cc:482] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:55.117439 22916 leader_election.cc:288] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:55.254278 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:55.113543 22931 raft_consensus.cc:482] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:54.991250 19290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:55.248781 23133 raft_consensus.cc:2781] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331 [term 8 FOLLOWER]: Leader election won for term 8
I20260807 03:39:55.249696 23149 raft_consensus.cc:686] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [term 6 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
W20260807 03:39:55.319895 18293 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57276 (request call id 1738) took 1860 ms (client timeout 1717 ms). Trace:
W20260807 03:39:55.320427 18293 rpcz_store.cc:269] 0807 03:39:53.459158 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:53.460242 (+ 1084us) service_pool.cc:227] Handling call
0807 03:39:55.319618 (+1859376us) spinlock_profiling.cc:243] Waited 988 ms on lock 0x7b5400302ad4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:55.319800 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2272151808}
I20260807 03:39:55.321667 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:55.322906 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:55.289021 23166 raft_consensus.cc:482] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5c9b5062bdfa46af9396f1a85025a331)
I20260807 03:39:55.326388 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:55.324014 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:55.335783 18293 raft_consensus.cc:3037] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:55.220427 18305 raft_consensus.cc:2379] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 8: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:55.266585 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:55.350591 18297 raft_consensus.cc:2454] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 7.
W20260807 03:39:55.352839 18079 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33254 (request call id 2367) took 1609 ms (client timeout 1875 ms). Trace:
I20260807 03:39:55.266688 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:54.994984 23136 leader_election.cc:288] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:54.994592 23130 leader_election.cc:288] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:55.267550 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:55.268384 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:54.993856 23001 raft_consensus.cc:2781] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader election won for term 9
I20260807 03:39:54.996415 22862 raft_consensus.cc:482] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:55.250545 18081 raft_consensus.cc:2454] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 8.
I20260807 03:39:55.201535 22896 raft_consensus.cc:2726] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260807 03:39:55.060469 22990 leader_election.cc:288] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:55.272594 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:55.280611 18191 raft_consensus.cc:2454] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 9.
W20260807 03:39:55.254033 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 309703us
I20260807 03:39:55.166918 22934 leader_election.cc:288] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:55.271288 22132 leader_election.cc:288] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:55.286347 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:55.294811 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:55.373883 18191 raft_consensus.cc:1261] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260807 03:39:55.116529 22980 leader_election.cc:288] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
W20260807 03:39:55.102931 23122 consensus_meta.cc:287] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331: Time spent flushing consensus metadata: real 0.764s user 0.009s sys 0.003s
I20260807 03:39:54.995891 23134 leader_election.cc:288] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:55.269814 22902 raft_consensus.cc:482] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:55.061295 23010 raft_consensus.cc:504] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:55.357520 18073 raft_consensus.cc:2454] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:39:55.360132 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:55.353369 18079 rpcz_store.cc:269] 0807 03:39:53.743353 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:53.743934 (+ 581us) service_pool.cc:227] Handling call
0807 03:39:55.352625 (+1608691us) spinlock_profiling.cc:243] Waited 79.6 ms on lock 0x7b540012e3d4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:55.352774 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":183107328}
I20260807 03:39:55.356791 18300 raft_consensus.cc:2454] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 8.
I20260807 03:39:55.297727 22931 raft_consensus.cc:3037] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:55.369726 22862 raft_consensus.cc:504] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:55.296924 19289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:55.378268 18080 raft_consensus.cc:2379] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 9: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:55.386572 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:55.270040 22865 raft_consensus.cc:504] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:55.325225 23166 raft_consensus.cc:504] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:55.388414 18141 leader_election.cc:302] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:55.416530 18141 leader_election.cc:302] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:55.418766 18141 leader_election.cc:302] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:55.420599 18141 leader_election.cc:302] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:54.992340 23167 consensus_queue.cc:1035] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.487s
I20260807 03:39:55.402714 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:55.444402 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:55.447319 18293 raft_consensus.cc:2454] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 9.
I20260807 03:39:55.377717 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:55.360653 23133 raft_consensus.cc:686] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331 [term 8 LEADER]: Becoming Leader. State: Replica: 5c9b5062bdfa46af9396f1a85025a331, State: Running, Role: LEADER
I20260807 03:39:55.408217 22857 leader_election.cc:288] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:55.420784 18072 raft_consensus.cc:2454] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 6.
I20260807 03:39:55.007719 23145 raft_consensus.cc:504] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:55.422495 18309 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57268 (request call id 2104) took 2401 ms (client timeout 1780 ms). Trace:
I20260807 03:39:55.423988 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:55.438741 18070 raft_consensus.cc:2454] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 9.
I20260807 03:39:55.439217 18082 raft_consensus.cc:3032] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 LEADER]: Stepping down as leader of term 7
W20260807 03:39:55.397204 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 457191us
I20260807 03:39:55.448808 18180 raft_consensus.cc:2454] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 6.
I20260807 03:39:55.450663 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:55.443066 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:55.395463 23001 raft_consensus.cc:686] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:55.422403 23055 raft_consensus.cc:2726] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:55.303834 22993 raft_consensus.cc:504] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:55.395900 23122 raft_consensus.cc:504] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:55.401961 22902 raft_consensus.cc:504] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:55.449762 18305 raft_consensus.cc:2379] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 8: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:55.522150 23149 consensus_queue.cc:237] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:55.439872 23134 raft_consensus.cc:2781] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 03:39:55.593364 18184 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44660 (request call id 2521) took 2604 ms. Trace:
I20260807 03:39:55.598201 23137 raft_consensus.cc:3032] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 LEADER]: Stepping down as leader of term 9
I20260807 03:39:55.592887 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:55.598785 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:55.449764 22882 raft_consensus.cc:3037] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Advancing to term 10
W20260807 03:39:55.600113 18027 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.660s user 0.017s sys 0.020s
I20260807 03:39:55.600471 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:55.439630 22530 raft_consensus.cc:2781] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260807 03:39:55.604141 18309 rpcz_store.cc:269] 0807 03:39:53.020744 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:53.066475 (+ 45731us) service_pool.cc:227] Handling call
0807 03:39:55.422345 (+2355870us) spinlock_profiling.cc:243] Waited 472 ms on lock 0x7b540035aa54. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:55.422433 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1085980032}
I20260807 03:39:55.607007 18082 raft_consensus.cc:729] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:55.615417 18075 raft_consensus.cc:3037] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:55.606954 18069 raft_consensus.cc:2454] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:39:55.960511 18069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33254 (request call id 2380) took 1219 ms. Trace:
I20260807 03:39:55.961048 18069 rpcz_store.cc:276] 0807 03:39:54.741135 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:54.932987 (+191852us) service_pool.cc:227] Handling call
0807 03:39:55.960447 (+1027460us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 03:39:55.962570 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:55.692467 22908 leader_election.cc:288] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:55.679873 23166 leader_election.cc:288] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
W20260807 03:39:55.915190 23116 consensus_meta.cc:287] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 0.694s user 0.010s sys 0.000s
W20260807 03:39:55.599485 22441 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.518s user 0.008s sys 0.017s
I20260807 03:39:55.606149 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:39:55.985183 18027 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.046s user 0.017s sys 0.028s
I20260807 03:39:55.964336 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:55.987790 18193 raft_consensus.cc:2379] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 9: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:55.973954 19295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:55.706650 18249 leader_election.cc:302] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:55.734155 17918 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35098 (request call id 88) took 4117 ms. Trace:
I20260807 03:39:55.989115 18189 raft_consensus.cc:2454] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 7.
I20260807 03:39:55.988608 18188 raft_consensus.cc:2454] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 8.
I20260807 03:39:55.760612 18070 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33254 (request call id 2381) took 1019 ms. Trace:
I20260807 03:39:55.611320 18248 leader_election.cc:302] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:55.990855 18070 rpcz_store.cc:276] 0807 03:39:54.741450 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:54.914708 (+173258us) service_pool.cc:227] Handling call
0807 03:39:55.760425 (+845717us) spinlock_profiling.cc:243] Waited 523 ms on lock 0x7b5400033c54. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:55.760549 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1202894208}
W20260807 03:39:55.601128 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 154319us
I20260807 03:39:55.865221 17915 catalog_manager.cc:5495] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 5 to 8, leader changed from d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 8 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260807 03:39:55.620090 18295 raft_consensus.cc:1261] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260807 03:39:55.608124 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:55.794749 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:55.608402 18073 raft_consensus.cc:2454] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 8.
I20260807 03:39:55.994055 18081 raft_consensus.cc:2454] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 6.
I20260807 03:39:55.612977 19240 leader_election.cc:302] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:55.612437 18293 raft_consensus.cc:1261] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260807 03:39:55.694248 19292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:55.601641 18080 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58950 (request call id 240) took 1504 ms. Trace:
I20260807 03:39:55.679580 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:55.615639 19297 raft_consensus.cc:1226] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331 [term 5 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20260807 03:39:55.997892 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 291339us
W20260807 03:39:55.999251 19240 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 386329us
I20260807 03:39:55.996977 18080 rpcz_store.cc:276] 0807 03:39:54.097647 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:55.422731 (+1325084us) service_pool.cc:227] Handling call
0807 03:39:55.601285 (+178554us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:55.601422 (+ 137us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:55.601439 (+ 17us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:55.601463 (+ 24us) raft_consensus.cc:1631] Marking committed up to 7
0807 03:39:55.601488 (+ 25us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:55.601528 (+ 40us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:55.601601 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 03:39:56.000329 18300 raft_consensus.cc:3037] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:55.604456 18072 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33268 (request call id 2039) took 1466 ms. Trace:
I20260807 03:39:55.995631 18293 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57268 (request call id 2115) took 1173 ms. Trace:
I20260807 03:39:56.001331 18072 rpcz_store.cc:276] 0807 03:39:54.137788 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:55.603645 (+1465857us) service_pool.cc:227] Handling call
0807 03:39:55.603945 (+ 300us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:55.604137 (+ 192us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:55.604166 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:55.604209 (+ 43us) raft_consensus.cc:1631] Marking committed up to 5
0807 03:39:55.604244 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:55.604301 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:55.604407 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 03:39:55.680887 18140 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.707s user 0.006s sys 0.005s
I20260807 03:39:55.993261 17915 catalog_manager.cc:5495] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: config changed from index -1 to 5, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New cstate: current_term: 6 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
W20260807 03:39:55.720186 18071 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33254 (request call id 2366) took 1981 ms (client timeout 2027 ms). Trace:
W20260807 03:39:56.035612 18140 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.062s user 0.011s sys 0.008s
I20260807 03:39:56.035887 17915 catalog_manager.cc:5495] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 9 to 10, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 10 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
W20260807 03:39:56.037114 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 1.13136e+06us
I20260807 03:39:55.986620 18082 consensus_queue.cc:260] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:56.036729 18071 rpcz_store.cc:269] 0807 03:39:53.738654 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:53.761715 (+ 23061us) service_pool.cc:227] Handling call
0807 03:39:55.719963 (+1958248us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b54003516d4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:55.720119 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2693339520}
I20260807 03:39:56.035023 18141 leader_election.cc:302] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
W20260807 03:39:55.606233 23117 log.cc:924] Time spent T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 1.577s user 0.005s sys 0.010s
W20260807 03:39:55.633229 23045 consensus_meta.cc:287] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.976s user 0.011s sys 0.004s
W20260807 03:39:55.600839 22658 consensus_meta.cc:287] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 0.956s user 0.010s sys 0.010s
W20260807 03:39:55.600822 23153 thread.cc:636] tablet-download-pool [worker] (thread pool) Time spent creating pthread: real 0.589s user 0.014s sys 0.025s
I20260807 03:39:56.088784 18071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33268 (request call id 2050) took 1598 ms. Trace:
I20260807 03:39:56.037590 18295 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57276 (request call id 1753) took 1231 ms. Trace:
W20260807 03:39:55.628729 23101 consensus_meta.cc:287] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 1.104s user 0.009s sys 0.008s
I20260807 03:39:56.045252 18141 leader_election.cc:302] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:55.629717 23127 leader_election.cc:288] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:56.040961 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:56.095618 18069 raft_consensus.cc:3037] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:56.084728 18081 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33268 (request call id 2043) took 1945 ms. Trace:
I20260807 03:39:55.985607 23134 raft_consensus.cc:482] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:55.605262 22888 raft_consensus.cc:3037] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:55.968237 19288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:56.038267 22916 raft_consensus.cc:2781] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader election won for term 9
W20260807 03:39:55.736581 23183 log.cc:924] Time spent T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.102s user 0.000s sys 0.000s
I20260807 03:39:55.984790 18184 rpcz_store.cc:276] 0807 03:39:52.988761 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:52.992644 (+ 3883us) service_pool.cc:227] Handling call
0807 03:39:52.992938 (+ 294us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:53.041723 (+ 48785us) raft_consensus.cc:1520] Early marking committed up to index 5
0807 03:39:53.216400 (+174677us) log.cc:842] Serialized 14 byte log entry
0807 03:39:53.402165 (+185765us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:53.402231 (+ 66us) raft_consensus.cc:1631] Marking committed up to 5
0807 03:39:53.402293 (+ 62us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:55.534845 (+2132552us) spinlock_profiling.cc:243] Waited 48.6 ms on lock 0x7b5400151154. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:39:55.593060 (+ 58215us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:55.593247 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":336484,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"raft.queue_time_us":801,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":3512,"spinlock_wait_cycles":111690112,"thread_start_us":356928,"threads_started":2,"wal-append.queue_time_us":512663,"wal-append.run_cpu_time_us":2091,"wal-append.run_wall_time_us":1330385}
I20260807 03:39:56.002785 22909 raft_consensus.cc:2726] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260807 03:39:55.605733 23145 leader_election.cc:288] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:55.615120 22917 raft_consensus.cc:482] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:56.189388 22916 raft_consensus.cc:686] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [term 9 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:55.631623 23015 leader_election.cc:288] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:55.989639 17918 rpcz_store.cc:276] 0807 03:39:51.616870 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:51.617010 (+ 140us) service_pool.cc:227] Handling call
0807 03:39:55.734083 (+4117073us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 03:39:53.349192 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 03:39:53.349781 (+ 589us) write_op.cc:432] Acquiring schema lock in shared mode
0807 03:39:53.349825 (+ 44us) write_op.cc:435] Acquired schema lock
0807 03:39:53.349837 (+ 12us) tablet.cc:660] Decoding operations
0807 03:39:53.350359 (+ 522us) write_op.cc:620] Acquiring the partition lock for write op
0807 03:39:53.350463 (+ 104us) write_op.cc:641] Partition lock acquired for write op
0807 03:39:53.350492 (+ 29us) tablet.cc:683] Acquiring locks for 1 operations
0807 03:39:53.350692 (+ 200us) tablet.cc:699] Row locks acquired
0807 03:39:53.350706 (+ 14us) write_op.cc:260] PREPARE: finished
0807 03:39:53.350859 (+ 153us) write_op.cc:270] Start()
0807 03:39:53.350971 (+ 112us) write_op.cc:276] Timestamp: P: 1786073993350801 usec, L: 0
0807 03:39:53.350987 (+ 16us) op_driver.cc:331] REPLICATION: starting
0807 03:39:53.351724 (+ 737us) log.cc:842] Serialized 603 byte log entry
0807 03:39:53.989042 (+637318us) op_driver.cc:444] REPLICATION: finished
0807 03:39:54.337462 (+348420us) write_op.cc:301] APPLY: starting
0807 03:39:54.337714 (+ 252us) tablet.cc:1365] starting BulkCheckPresence
0807 03:39:54.338017 (+ 303us) tablet.cc:1368] finished BulkCheckPresence
0807 03:39:54.338248 (+ 231us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 03:39:54.338700 (+ 452us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 03:39:54.338926 (+ 226us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0807 03:39:54.338976 (+ 50us) write_op.cc:312] APPLY: finished
0807 03:39:54.339283 (+ 307us) log.cc:842] Serialized 23 byte log entry
0807 03:39:54.339439 (+ 156us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:39:54.339531 (+ 92us) write_op.cc:454] Released schema lock
0807 03:39:54.339573 (+ 42us) write_op.cc:341] FINISH: Updating metrics
0807 03:39:54.339929 (+ 356us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:39:54.339959 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":894104,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":314222,"apply.run_cpu_time_us":2820,"apply.run_wall_time_us":12173,"num_ops":1,"prepare.queue_time_us":317239,"prepare.run_cpu_time_us":30370,"prepare.run_wall_time_us":113660,"raft.queue_time_us":395576,"raft.run_cpu_time_us":37748,"raft.run_wall_time_us":244174,"replication_time_us":633062,"thread_start_us":857226,"threads_started":4,"wal-append.queue_time_us":208547,"wal-append.run_cpu_time_us":33589,"wal-append.run_wall_time_us":1878681}]]}
I20260807 03:39:56.038668 18082 raft_consensus.cc:3037] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:55.734357 22931 raft_consensus.cc:504] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:55.976064 22865 leader_election.cc:288] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:55.604753 23041 leader_election.cc:288] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:56.093202 18080 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33268 (request call id 2049) took 1789 ms. Trace:
I20260807 03:39:56.039927 17915 catalog_manager.cc:5495] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 2 to 3, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 3 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } }
W20260807 03:39:55.985242 22441 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.903s user 0.014s sys 0.023s
I20260807 03:39:55.628731 19291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:56.084261 18070 raft_consensus.cc:1261] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260807 03:39:55.614122 22929 raft_consensus.cc:482] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:55.616376 23122 leader_election.cc:288] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:55.978462 23010 leader_election.cc:288] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:55.706347 19290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:55.912567 19289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:56.085636 18072 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58950 (request call id 241) took 1726 ms. Trace:
W20260807 03:39:56.097136 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 52000.6us
I20260807 03:39:56.189707 22917 raft_consensus.cc:504] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:56.211381 18141 leader_election.cc:302] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:55.615851 23124 leader_election.cc:288] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:56.097347 18073 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33268 (request call id 2042) took 1958 ms. Trace:
I20260807 03:39:56.089479 18071 rpcz_store.cc:276] 0807 03:39:54.490701 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:56.040307 (+1549606us) service_pool.cc:227] Handling call
0807 03:39:56.086672 (+ 46365us) spinlock_profiling.cc:243] Waited 1.06 ms on lock 0x7b5800001298. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f25f851 00007fe98f25f702 00007fe99814fb09 00007fe99708b730 00007fe99715179b 00007fe997151821 00007fe9971152b0 00007fe9971007f9 00007fe9971002a2 00007fe9907e6684
0807 03:39:56.088166 (+ 1494us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:56.088387 (+ 221us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:56.088424 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:56.088480 (+ 56us) raft_consensus.cc:1631] Marking committed up to 5
0807 03:39:56.088525 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:56.088594 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:56.088715 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2447744}
I20260807 03:39:55.633136 22902 leader_election.cc:288] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:55.985070 22530 raft_consensus.cc:482] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:55.613411 22993 leader_election.cc:288] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:56.100131 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:55.604871 22862 leader_election.cc:288] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:55.973351 23017 leader_election.cc:288] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:56.096086 18081 rpcz_store.cc:276] 0807 03:39:54.138840 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:56.008112 (+1869272us) service_pool.cc:227] Handling call
0807 03:39:56.084282 (+ 76170us) spinlock_profiling.cc:243] Waited 33.1 ms on lock 0x7b5800001298. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f25f851 00007fe98f25f702 00007fe99814fb09 00007fe99708b730 00007fe99715179b 00007fe997151821 00007fe9971152b0 00007fe9971007f9 00007fe9971002a2 00007fe9907e6684
0807 03:39:56.084417 (+ 135us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:56.084539 (+ 122us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:56.084553 (+ 14us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:56.084576 (+ 23us) raft_consensus.cc:1631] Marking committed up to 4
0807 03:39:56.084593 (+ 17us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:56.084622 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:56.084686 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":76156672}
I20260807 03:39:56.038092 22426 raft_consensus.cc:482] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:56.039121 18249 leader_election.cc:398] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Vote denied by peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057) with higher term. Message: Invalid argument: T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 5. Current term is 6.
I20260807 03:39:56.223075 18249 leader_election.cc:401] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
W20260807 03:39:56.087383 23153 thread.cc:602] tablet-download-pool [worker] (thread pool) Time spent starting thread: real 1.084s user 0.017s sys 0.038s
I20260807 03:39:56.089871 18295 rpcz_store.cc:276] 0807 03:39:54.806378 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:55.423028 (+616650us) service_pool.cc:227] Handling call
0807 03:39:55.619789 (+196761us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:55.992990 (+373201us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:56.037422 (+ 44432us) spinlock_profiling.cc:243] Waited 150 ms on lock 0x7b54001c3c50. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bea09 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:56.037536 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":344870144}
I20260807 03:39:55.615355 22896 consensus_queue.cc:1035] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.480s
I20260807 03:39:55.612672 23053 consensus_queue.cc:1035] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.482s
I20260807 03:39:56.176453 18069 raft_consensus.cc:2454] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 5.
I20260807 03:39:56.226771 22980 raft_consensus.cc:3037] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:56.228645 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:56.229386 18069 raft_consensus.cc:2454] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 9.
I20260807 03:39:56.230158 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:56.222859 22476 raft_consensus.cc:2726] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 03:39:56.170641 18075 raft_consensus.cc:2454] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 9.
I20260807 03:39:56.181444 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:56.181771 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 570463us
I20260807 03:39:55.985978 23137 raft_consensus.cc:729] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:56.182072 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:56.197547 22916 consensus_queue.cc:237] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:55.632725 23013 raft_consensus.cc:504] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:55.964540 23116 raft_consensus.cc:504] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:56.185653 23132 raft_consensus.cc:2726] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 03:39:56.093065 22897 raft_consensus.cc:2781] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:39:56.042902 23040 consensus_queue.cc:1035] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.129s
I20260807 03:39:56.249521 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:56.190977 18185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50060 (request call id 194) took 1648 ms. Trace:
I20260807 03:39:56.169921 23134 raft_consensus.cc:3037] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:56.095387 22847 raft_consensus.cc:482] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:55.615226 23158 leader_election.cc:288] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:56.163008 22984 raft_consensus.cc:482] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:56.090623 19287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:56.200809 19297 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59906 (request call id 223) took 1878 ms. Trace:
I20260807 03:39:55.698081 23133 consensus_queue.cc:237] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:56.259754 22847 raft_consensus.cc:504] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:56.088347 22658 raft_consensus.cc:504] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:56.203948 18070 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33268 (request call id 2041) took 2065 ms. Trace:
I20260807 03:39:56.268432 18070 rpcz_store.cc:276] 0807 03:39:54.138304 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:56.010569 (+1872265us) service_pool.cc:227] Handling call
0807 03:39:56.083699 (+ 73130us) spinlock_profiling.cc:243] Waited 30 ms on lock 0x7b5800001298. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f25f851 00007fe98f25f702 00007fe99814fb09 00007fe99708b730 00007fe99715179b 00007fe997151821 00007fe9971152b0 00007fe9971007f9 00007fe9971002a2 00007fe9907e6684
0807 03:39:56.083909 (+ 210us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:56.203729 (+119820us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:56.203898 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":69007744}
I20260807 03:39:56.203125 18080 rpcz_store.cc:276] 0807 03:39:54.303656 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:56.039163 (+1735507us) service_pool.cc:227] Handling call
0807 03:39:56.092859 (+ 53696us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:56.093007 (+ 148us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:56.093022 (+ 15us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:56.093046 (+ 24us) raft_consensus.cc:1631] Marking committed up to 3
0807 03:39:56.093064 (+ 18us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:56.093094 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:56.093159 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 03:39:56.269027 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:56.088217 23045 raft_consensus.cc:504] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:55.609339 23001 consensus_queue.cc:237] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:56.274044 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:56.211589 17915 catalog_manager.cc:5495] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 5 to 6, leader changed from d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 6 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:39:56.212653 18073 rpcz_store.cc:276] 0807 03:39:54.138579 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:56.036836 (+1898257us) service_pool.cc:227] Handling call
0807 03:39:56.096857 (+ 60021us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:56.097024 (+ 167us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:56.097046 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:56.097080 (+ 34us) raft_consensus.cc:1631] Marking committed up to 3
0807 03:39:56.097166 (+ 86us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:56.097218 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:56.097300 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 03:39:56.213167 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:56.210826 18072 rpcz_store.cc:276] 0807 03:39:54.359095 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:56.038950 (+1679855us) service_pool.cc:227] Handling call
0807 03:39:56.084905 (+ 45955us) spinlock_profiling.cc:243] Waited 2.29 ms on lock 0x7b5800001298. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f25f851 00007fe98f25f702 00007fe99814fb09 00007fe99708b730 00007fe99715179b 00007fe997151821 00007fe9971152b0 00007fe9971007f9 00007fe9971002a2 00007fe9907e6684
0807 03:39:56.085142 (+ 237us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:56.085352 (+ 210us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:56.085387 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:56.085428 (+ 41us) raft_consensus.cc:1631] Marking committed up to 5
0807 03:39:56.085458 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:56.085504 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:56.085589 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5259520}
I20260807 03:39:56.275581 18071 raft_consensus.cc:3037] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:56.214282 18305 raft_consensus.cc:1226] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 8. Current term is 9. Ops: []
I20260807 03:39:56.195932 23145 raft_consensus.cc:2726] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260807 03:39:56.100205 23101 raft_consensus.cc:504] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:56.216256 22530 raft_consensus.cc:3037] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:56.181692 22427 consensus_queue.cc:1035] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.028s
I20260807 03:39:56.217941 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:56.220484 18141 leader_election.cc:302] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:56.222065 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
W20260807 03:39:56.225188 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 225605us
I20260807 03:39:56.229624 19239 leader_election.cc:302] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:56.290406 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:56.290871 19239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 61361.2us
I20260807 03:39:56.291114 18312 raft_consensus.cc:2379] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 10: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:56.230795 18295 raft_consensus.cc:2454] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 9.
I20260807 03:39:56.234129 18248 leader_election.cc:302] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
W20260807 03:39:56.292557 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 58535.1us
I20260807 03:39:56.001989 18293 rpcz_store.cc:276] 0807 03:39:54.822480 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:55.447903 (+625423us) service_pool.cc:227] Handling call
0807 03:39:55.449314 (+ 1411us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:55.995387 (+546073us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:55.995579 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 03:39:55.628839 22921 raft_consensus.cc:504] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:56.210752 22929 raft_consensus.cc:504] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:56.237648 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:56.247879 18300 raft_consensus.cc:2454] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 9.
I20260807 03:39:56.250823 18185 rpcz_store.cc:276] 0807 03:39:54.542584 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:54.717781 (+175197us) service_pool.cc:227] Handling call
0807 03:39:54.718063 (+ 282us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:54.718375 (+ 312us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:54.718400 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:39:54.738364 (+ 19964us) log.cc:842] Serialized 26 byte log entry
0807 03:39:55.101791 (+363427us) raft_consensus.cc:1631] Marking committed up to 6
0807 03:39:55.101831 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:55.101904 (+ 73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:56.190719 (+1088815us) raft_consensus.cc:1671] finished
0807 03:39:56.190744 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:56.190909 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":87,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":6934,"thread_start_us":314803,"threads_started":1,"wal-append.queue_time_us":869076}
I20260807 03:39:56.225131 22426 raft_consensus.cc:504] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:56.299523 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:56.235953 23153 tablet_copy_client.cc:316] T d416e1015a2644958b0c8f9147d403a2 P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Beginning tablet copy session from remote peer at address 127.15.198.131:33021
I20260807 03:39:56.260782 18075 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33254 (request call id 2386) took 1514 ms. Trace:
I20260807 03:39:56.249027 22897 raft_consensus.cc:482] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:56.263960 19297 rpcz_store.cc:276] 0807 03:39:54.315645 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:54.316587 (+ 942us) service_pool.cc:227] Handling call
0807 03:39:54.316880 (+ 293us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:55.999152 (+1682272us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:56.193699 (+194547us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b540078f5d4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:39:56.193868 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2986256640}
I20260807 03:39:56.207581 18082 raft_consensus.cc:2454] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 8.
I20260807 03:39:56.262205 22984 raft_consensus.cc:504] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:56.271553 18183 raft_consensus.cc:1226] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 6. Current term is 7. Ops: []
I20260807 03:39:56.274600 18070 raft_consensus.cc:3037] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:56.278950 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:56.281385 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:56.283521 17915 catalog_manager.cc:5495] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 5 to 6, leader changed from d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 6 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
W20260807 03:39:56.293555 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 73193.7us
I20260807 03:39:56.293813 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:56.294654 18249 leader_election.cc:302] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:56.296355 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
W20260807 03:39:56.296645 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 285071us
I20260807 03:39:56.297441 18248 leader_election.cc:302] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:56.298565 18079 raft_consensus.cc:2379] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 6: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:56.298696 18069 raft_consensus.cc:2454] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 8.
I20260807 03:39:56.300989 18071 raft_consensus.cc:2454] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 9.
I20260807 03:39:56.310920 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:56.304854 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:56.308175 18082 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33268 (request call id 2028) took 3073 ms (client timeout 1980 ms). Trace:
I20260807 03:39:56.300639 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:56.303354 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:56.351052 18082 rpcz_store.cc:269] 0807 03:39:53.234595 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:53.234715 (+ 120us) service_pool.cc:227] Handling call
0807 03:39:55.439080 (+2204365us) spinlock_profiling.cc:243] Waited 143 ms on lock 0x7b5400181da0. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe99214f730 00007fe9921c2eff 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86
0807 03:39:56.308080 (+869000us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":329527680}
I20260807 03:39:56.269976 18310 raft_consensus.cc:2454] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 4.
W20260807 03:39:56.352720 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 58104.7us
I20260807 03:39:56.307758 18075 rpcz_store.cc:276] 0807 03:39:54.746708 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:55.246357 (+499649us) service_pool.cc:227] Handling call
0807 03:39:56.260581 (+1014224us) spinlock_profiling.cc:243] Waited 243 ms on lock 0x7b540037d554. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:56.260713 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":558040320}
I20260807 03:39:56.296087 23145 raft_consensus.cc:2726] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260807 03:39:56.347538 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 50169.7us
I20260807 03:39:56.358029 18248 leader_election.cc:302] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:56.353873 18249 leader_election.cc:302] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:39:56.311692 18189 raft_consensus.cc:1226] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:56.351765 18300 raft_consensus.cc:2454] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 8.
I20260807 03:39:56.354526 18080 raft_consensus.cc:1226] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260807 03:39:56.354560 18075 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33254 (request call id 2389) took 1590 ms. Trace:
I20260807 03:39:56.359097 18075 rpcz_store.cc:276] 0807 03:39:54.764474 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:56.353819 (+1589345us) service_pool.cc:227] Handling call
0807 03:39:56.354099 (+ 280us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:56.354269 (+ 170us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:56.354292 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:56.354330 (+ 38us) raft_consensus.cc:1631] Marking committed up to 3
0807 03:39:56.354359 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:56.354427 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:56.354514 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 03:39:56.354919 18140 connection.cc:232] RPC call timeout handler was delayed by -1.1170214160811156s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.198s for a response
I20260807 03:39:56.350687 18309 raft_consensus.cc:2454] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 5.
I20260807 03:39:56.346566 18073 raft_consensus.cc:3037] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:56.350135 18071 raft_consensus.cc:1261] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260807 03:39:56.295083 23045 leader_election.cc:288] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:56.356375 18305 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc"
dest_uuid: "66a36e1ac8e945e893668e282d394c49"
mode: GRACEFUL
new_leader_uuid: "3c4bceb836db4536a0f255fdf86abc02"
from {username='slave'} at 127.0.0.1:58736
I20260807 03:39:56.356622 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:56.423069 18305 raft_consensus.cc:595] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 8 LEADER]: Received request to transfer leadership to TS 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:56.364512 18248 leader_election.cc:302] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:56.358728 18070 raft_consensus.cc:2454] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 8.
I20260807 03:39:56.424437 18080 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33254 (request call id 2384) took 1678 ms. Trace:
I20260807 03:39:56.416579 18309 raft_consensus.cc:1226] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 9. Current term is 10. Ops: []
I20260807 03:39:56.416481 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:56.417541 18180 raft_consensus.cc:1226] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 7. Current term is 8. Ops: []
I20260807 03:39:56.426091 18184 raft_consensus.cc:2379] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 9: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:56.417870 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:56.426476 18312 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57268 (request call id 2124) took 1198 ms (client timeout 1594 ms). Trace:
W20260807 03:39:56.418440 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 58924.8us
W20260807 03:39:56.426822 18312 rpcz_store.cc:269] 0807 03:39:55.228454 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:55.293594 (+ 65140us) service_pool.cc:227] Handling call
0807 03:39:56.426345 (+1132751us) spinlock_profiling.cc:243] Waited 960 ms on lock 0x7b540035b454. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:56.426433 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2208986496}
I20260807 03:39:56.427202 18027 leader_election.cc:302] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:56.428040 18027 leader_election.cc:302] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:56.295528 23140 raft_consensus.cc:2781] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 03:39:56.416373 18185 raft_consensus.cc:3037] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:56.361543 23166 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:56.435212 18071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33254 (request call id 2378) took 1694 ms. Trace:
I20260807 03:39:56.451962 18071 rpcz_store.cc:276] 0807 03:39:54.740489 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:56.349540 (+1609051us) service_pool.cc:227] Handling call
0807 03:39:56.349825 (+ 285us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:56.434905 (+ 85080us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:56.435151 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 03:39:56.453655 18071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33254 (request call id 2397) took 1624 ms. Trace:
I20260807 03:39:56.454049 18071 rpcz_store.cc:276] 0807 03:39:54.829580 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:56.452856 (+1623276us) service_pool.cc:227] Handling call
0807 03:39:56.453164 (+ 308us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:56.453340 (+ 176us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:56.453366 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:56.453411 (+ 45us) raft_consensus.cc:1631] Marking committed up to 5
0807 03:39:56.453455 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:56.453509 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:56.453605 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 03:39:56.429251 18141 leader_election.cc:302] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:56.430627 18069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33254 (request call id 2395) took 1610 ms. Trace:
I20260807 03:39:56.457206 18069 rpcz_store.cc:276] 0807 03:39:54.819955 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:56.415309 (+1595354us) service_pool.cc:227] Handling call
0807 03:39:56.430095 (+ 14786us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:56.430320 (+ 225us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:56.430354 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:56.430400 (+ 46us) raft_consensus.cc:1631] Marking committed up to 3
0807 03:39:56.430443 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:56.430496 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:56.430579 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 03:39:56.310134 22932 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 8 current term: 8 is higher.
I20260807 03:39:56.348713 17915 catalog_manager.cc:5495] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 11 to 13, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 13 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:56.431574 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:56.432446 18027 leader_election.cc:302] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:56.434599 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:56.310582 22897 raft_consensus.cc:3037] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:56.497170 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:56.497080 17915 catalog_manager.cc:5495] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 5 to 7, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 7 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:56.498289 18180 raft_consensus.cc:2454] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 10.
I20260807 03:39:56.495348 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:56.499037 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:56.494994 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:56.460278 18071 raft_consensus.cc:1226] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Rejecting Update request from peer 5c9b5062bdfa46af9396f1a85025a331 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260807 03:39:56.347400 22658 leader_election.cc:288] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:56.461741 18297 raft_consensus.cc:2454] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 10.
I20260807 03:39:56.464254 18069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58950 (request call id 244) took 1284 ms. Trace:
I20260807 03:39:56.461442 18075 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33254 (request call id 2394) took 1647 ms. Trace:
W20260807 03:39:56.462596 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 115775us
I20260807 03:39:56.352017 23017 raft_consensus.cc:482] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:56.347910 22931 leader_election.cc:288] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:56.500085 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:56.431241 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:56.507006 18180 raft_consensus.cc:2379] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 9: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:56.500846 18075 rpcz_store.cc:276] 0807 03:39:54.814183 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:56.360025 (+1545842us) service_pool.cc:227] Handling call
0807 03:39:56.461041 (+101016us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:56.461183 (+ 142us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:56.461205 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:56.461238 (+ 33us) raft_consensus.cc:1631] Marking committed up to 5
0807 03:39:56.461266 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:56.461308 (+ 42us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:56.461392 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 03:39:56.361904 23132 raft_consensus.cc:2781] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:39:56.361654 23021 raft_consensus.cc:2781] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:56.501691 18141 leader_election.cc:302] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:56.361382 22762 raft_consensus.cc:2781] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260807 03:39:56.514117 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 81751.7us
I20260807 03:39:56.515512 18141 leader_election.cc:302] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:39:56.494623 18183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44660 (request call id 2563) took 1541 ms. Trace:
I20260807 03:39:56.507700 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:56.425724 18080 rpcz_store.cc:276] 0807 03:39:54.746213 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:56.353743 (+1607530us) service_pool.cc:227] Handling call
0807 03:39:56.354015 (+ 272us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:56.415014 (+ 60999us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:56.424383 (+ 9369us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 03:39:56.499782 18191 raft_consensus.cc:2454] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 9.
I20260807 03:39:56.508682 17915 catalog_manager.cc:5495] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 3 to 4, leader changed from 66a36e1ac8e945e893668e282d394c49 (127.15.198.131) to 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130). New cstate: current_term: 4 leader_uuid: "3c4bceb836db4536a0f255fdf86abc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 03:39:56.519737 18080 raft_consensus.cc:1261] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 03:39:56.520489 18080 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33254 (request call id 2401) took 1249 ms. Trace:
I20260807 03:39:56.521050 17915 catalog_manager.cc:5495] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 reported cstate change: term changed from 4 to 5, leader changed from 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130) to 66a36e1ac8e945e893668e282d394c49 (127.15.198.131). New cstate: current_term: 5 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } }
I20260807 03:39:56.497750 18184 raft_consensus.cc:2454] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 5.
I20260807 03:39:56.357030 22862 raft_consensus.cc:482] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:56.530613 23101 leader_election.cc:288] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:56.505582 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:56.516460 18183 rpcz_store.cc:276] 0807 03:39:54.953514 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:55.370313 (+416799us) service_pool.cc:227] Handling call
0807 03:39:55.370596 (+ 283us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:56.352204 (+981608us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:56.494329 (+142125us) spinlock_profiling.cc:243] Waited 901 ms on lock 0x7b54001df954. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:39:56.494553 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2071510784}
I20260807 03:39:56.510360 18071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58950 (request call id 243) took 1331 ms. Trace:
I20260807 03:39:56.520854 18080 rpcz_store.cc:276] 0807 03:39:55.271381 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:56.519019 (+1247638us) service_pool.cc:227] Handling call
0807 03:39:56.519345 (+ 326us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:56.520230 (+ 885us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:56.520430 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 03:39:56.500802 18069 rpcz_store.cc:276] 0807 03:39:55.179841 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:56.458218 (+1278377us) service_pool.cc:227] Handling call
0807 03:39:56.463689 (+ 5471us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:56.463876 (+ 187us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:56.463911 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:56.463959 (+ 48us) raft_consensus.cc:1631] Marking committed up to 5
0807 03:39:56.464000 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:56.464061 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:56.464194 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 03:39:56.492530 23166 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:56.529544 23041 raft_consensus.cc:2726] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:56.530076 18027 leader_election.cc:302] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:56.541091 18027 leader_election.cc:302] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:56.530726 22862 raft_consensus.cc:504] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:56.417987 23124 raft_consensus.cc:482] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:56.543414 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:56.432578 23010 raft_consensus.cc:2741] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260807 03:39:56.537844 18141 leader_election.cc:302] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:56.311388 22921 leader_election.cc:288] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:56.538837 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:56.547605 18187 raft_consensus.cc:2454] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 8.
I20260807 03:39:56.528360 18073 raft_consensus.cc:2454] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 6.
I20260807 03:39:56.451442 22902 raft_consensus.cc:2781] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260807 03:39:56.553508 18027 leader_election.cc:302] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:56.451215 23109 raft_consensus.cc:2781] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:39:56.557222 23109 raft_consensus.cc:482] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:56.505725 23140 raft_consensus.cc:686] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331 [term 5 LEADER]: Becoming Leader. State: Replica: 5c9b5062bdfa46af9396f1a85025a331, State: Running, Role: LEADER
I20260807 03:39:56.505231 23040 raft_consensus.cc:2726] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260807 03:39:56.541790 22931 raft_consensus.cc:2781] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 03:39:56.541224 23145 raft_consensus.cc:2781] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 03:39:56.418154 22985 consensus_queue.cc:1046] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.124s
I20260807 03:39:56.565918 18071 rpcz_store.cc:276] 0807 03:39:55.179385 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:56.454988 (+1275603us) service_pool.cc:227] Handling call
0807 03:39:56.460000 (+ 5012us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:56.510036 (+ 50036us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:56.510300 (+ 264us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 03:39:56.559257 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24d0bbea8bd49d482ad95538ca6e631" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:56.535486 23101 consensus_queue.cc:1035] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20260807 03:39:56.464219 23045 raft_consensus.cc:2726] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 03:39:56.512890 22917 leader_election.cc:288] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:56.556025 22902 raft_consensus.cc:482] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:56.558139 23041 raft_consensus.cc:2726] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:56.561554 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:56.566978 22929 leader_election.cc:288] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:56.536172 22658 raft_consensus.cc:2781] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 03:39:56.533718 22762 raft_consensus.cc:482] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:56.554463 18185 raft_consensus.cc:1261] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260807 03:39:56.560921 18027 leader_election.cc:302] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:56.355495 23137 consensus_queue.cc:260] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:56.576619 18025 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.132:41381, user_credentials={real_user=slave}} blocked reactor thread for 43983.4us
I20260807 03:39:56.577379 23166 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:56.570250 23010 raft_consensus.cc:2781] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:56.568084 23015 consensus_queue.cc:1046] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.014s
I20260807 03:39:56.582451 22862 leader_election.cc:288] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:56.584301 18079 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33254 (request call id 2391) took 1818 ms. Trace:
I20260807 03:39:56.584609 19297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:56.572116 18069 raft_consensus.cc:1261] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260807 03:39:56.625730 18079 rpcz_store.cc:276] 0807 03:39:54.766647 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:55.404919 (+638272us) service_pool.cc:227] Handling call
0807 03:39:56.584045 (+1179126us) spinlock_profiling.cc:243] Waited 690 ms on lock 0x7b54002c47d4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:56.584250 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1586009856}
I20260807 03:39:56.626251 18069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33254 (request call id 2404) took 1006 ms. Trace:
I20260807 03:39:56.574518 22931 raft_consensus.cc:482] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:56.552752 23013 leader_election.cc:288] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:56.571085 23199 log.cc:924] Time spent T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.224s user 0.000s sys 0.001s
I20260807 03:39:56.534200 23017 raft_consensus.cc:504] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:56.560529 22865 raft_consensus.cc:2781] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260807 03:39:56.582867 22847 leader_election.cc:288] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:56.597401 18140 leader_election.cc:302] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 5c9b5062bdfa46af9396f1a85025a331
W20260807 03:39:56.644063 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 46778.2us
I20260807 03:39:56.645205 18140 leader_election.cc:302] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:56.598805 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e3ecc1bf7d4ca9a7096616efc98585" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:56.626570 18069 rpcz_store.cc:276] 0807 03:39:55.619814 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:56.556328 (+936514us) service_pool.cc:227] Handling call
0807 03:39:56.569833 (+ 13505us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:56.626030 (+ 56197us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:56.626203 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 03:39:56.647298 18187 raft_consensus.cc:2379] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 6: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:56.569286 23109 raft_consensus.cc:3037] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:56.511741 23132 raft_consensus.cc:482] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:56.603488 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:56.628489 18070 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33254 (request call id 2396) took 1802 ms. Trace:
I20260807 03:39:56.573793 23145 raft_consensus.cc:482] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 5c9b5062bdfa46af9396f1a85025a331)
I20260807 03:39:56.559973 18141 leader_election.cc:302] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:56.654512 18070 rpcz_store.cc:276] 0807 03:39:54.825980 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:56.424909 (+1598929us) service_pool.cc:227] Handling call
0807 03:39:56.425350 (+ 441us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:56.543079 (+117729us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:56.543106 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:56.543146 (+ 40us) raft_consensus.cc:1631] Marking committed up to 9
0807 03:39:56.543181 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:56.628297 (+ 85116us) spinlock_profiling.cc:243] Waited 118 ms on lock 0x7b54002c9054. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:39:56.628337 (+ 40us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:56.628440 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":270261632}
W20260807 03:39:56.655470 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 95601us
I20260807 03:39:56.656031 22896 raft_consensus.cc:2781] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader election won for term 9
I20260807 03:39:56.656298 22881 raft_consensus.cc:2781] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260807 03:39:56.656570 22896 raft_consensus.cc:686] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:56.656677 22881 raft_consensus.cc:482] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:56.658358 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:56.632205 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 71382.5us
I20260807 03:39:56.554926 23124 raft_consensus.cc:504] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:56.635591 22862 raft_consensus.cc:2781] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 03:39:56.666064 22862 raft_consensus.cc:482] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:56.666605 22862 raft_consensus.cc:3037] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:56.669342 23021 raft_consensus.cc:482] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:56.669778 23021 raft_consensus.cc:3037] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:56.629040 23010 raft_consensus.cc:482] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:56.637835 18079 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:33238 (request call id 237) took 2266 ms. Trace:
I20260807 03:39:56.569296 22921 consensus_queue.cc:1035] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.058s
I20260807 03:39:56.622011 22985 raft_consensus.cc:3032] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 03:39:56.652267 18140 leader_election.cc:302] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:56.597851 22902 raft_consensus.cc:3037] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Advancing to term 10
I20260807 03:39:56.622534 22658 raft_consensus.cc:482] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:56.635896 23015 raft_consensus.cc:3032] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 LEADER]: Stepping down as leader of term 8
W20260807 03:39:56.697618 18298 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.603s user 0.046s sys 0.060s
I20260807 03:39:56.660542 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:56.634899 22931 raft_consensus.cc:3037] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:56.629360 22441 raft_consensus.cc:2726] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 03:39:56.658974 18305 raft_consensus.cc:2454] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 8.
I20260807 03:39:56.747570 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:56.663946 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:56.665439 22426 leader_election.cc:288] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:56.664930 22896 consensus_queue.cc:237] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:56.663609 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:56.638800 23137 raft_consensus.cc:3037] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Advancing to term 10
I20260807 03:39:56.657075 22881 raft_consensus.cc:3037] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Advancing to term 10
I20260807 03:39:56.631888 22762 raft_consensus.cc:3037] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:56.692832 18073 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4ba8413916634e039111c838fbe1e587"
dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
mode: GRACEFUL
new_leader_uuid: "66a36e1ac8e945e893668e282d394c49"
from {username='slave'} at 127.0.0.1:55742
I20260807 03:39:56.678421 18079 rpcz_store.cc:276] 0807 03:39:54.395566 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:56.636308 (+2240742us) service_pool.cc:227] Handling call
0807 03:39:56.637740 (+ 1432us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 03:39:56.694195 22985 raft_consensus.cc:729] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:56.643108 22865 raft_consensus.cc:482] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
W20260807 03:39:56.724025 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 71909.2us
I20260807 03:39:56.659864 18027 leader_election.cc:302] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:56.725207 18190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44662 (request call id 1703) took 2140 ms. Trace:
W20260807 03:39:56.762617 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 102825us
I20260807 03:39:56.653447 23040 raft_consensus.cc:2781] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260807 03:39:56.657353 23132 raft_consensus.cc:3037] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:56.660391 23145 raft_consensus.cc:3037] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:56.637871 22909 consensus_queue.cc:1046] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.058s
I20260807 03:39:56.662140 22984 leader_election.cc:288] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:56.734726 18080 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33254 (request call id 2407) took 1114 ms. Trace:
I20260807 03:39:56.769425 23017 leader_election.cc:288] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:56.769549 18080 rpcz_store.cc:276] 0807 03:39:55.620744 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:56.554130 (+933386us) service_pool.cc:227] Handling call
0807 03:39:56.554447 (+ 317us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:56.572162 (+ 17715us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:56.572204 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:56.572253 (+ 49us) raft_consensus.cc:1631] Marking committed up to 4
0807 03:39:56.572305 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:56.734449 (+162144us) spinlock_profiling.cc:243] Waited 17.5 ms on lock 0x7b5400290ed4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:39:56.734508 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:56.734657 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":40190720}
I20260807 03:39:56.723721 23015 raft_consensus.cc:729] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
W20260807 03:39:56.737131 18298 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.643s user 0.050s sys 0.060s
I20260807 03:39:56.676608 22427 raft_consensus.cc:482] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:56.689991 23010 raft_consensus.cc:3037] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:56.749316 18141 leader_election.cc:302] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:56.749641 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:56.724692 18072 raft_consensus.cc:2932] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Committing config change with OpId 6.5: config changed from index -1 to 5, NON_VOTER 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } } }
I20260807 03:39:56.765527 23040 raft_consensus.cc:482] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 5c9b5062bdfa46af9396f1a85025a331)
I20260807 03:39:56.777952 23040 raft_consensus.cc:3037] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 11 FOLLOWER]: Advancing to term 12
I20260807 03:39:56.777973 18027 leader_election.cc:302] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:56.781091 23017 raft_consensus.cc:2781] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260807 03:39:56.781463 23017 raft_consensus.cc:482] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:56.781783 23017 raft_consensus.cc:3037] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Advancing to term 11
I20260807 03:39:56.787703 22476 raft_consensus.cc:2781] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 03:39:56.788138 22476 raft_consensus.cc:482] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:56.788479 22476 raft_consensus.cc:3037] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:56.767642 22865 raft_consensus.cc:3037] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Advancing to term 10
I20260807 03:39:56.758013 18073 raft_consensus.cc:595] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Received request to transfer leadership to TS 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:56.723569 22658 raft_consensus.cc:3037] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Advancing to term 9
W20260807 03:39:56.751019 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 397168us
I20260807 03:39:56.762462 18190 rpcz_store.cc:276] 0807 03:39:54.586024 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:54.736238 (+150214us) service_pool.cc:227] Handling call
0807 03:39:54.736511 (+ 273us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:54.760001 (+ 23490us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:54.760030 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:39:55.609870 (+849840us) spinlock_profiling.cc:243] Waited 4.52 ms on lock 0x7b180046fea0. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe9981368b1 00007fe998143661 00007fe98f264f96 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe990ba2e11 00007fe990bb2442 00007fe9921fac11 00007fe9921bf58a
0807 03:39:55.610970 (+ 1100us) spinlock_profiling.cc:243] Waited 5.98 ms on lock 0x7b2c00047e58. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe990ba2e11 00007fe990bb2442 00007fe9921fac11 00007fe9921bf58a 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772
0807 03:39:55.611417 (+ 447us) log.cc:842] Serialized 26 byte log entry
0807 03:39:56.092647 (+481230us) raft_consensus.cc:1631] Marking committed up to 4
0807 03:39:56.092682 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:56.092741 (+ 59us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:56.724975 (+632234us) raft_consensus.cc:1671] finished
0807 03:39:56.724998 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:56.725150 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22961,"raft.queue_time_us":30879,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":33084,"spinlock_wait_cycles":24136704,"thread_start_us":480901,"threads_started":1,"wal-append.queue_time_us":650520}
I20260807 03:39:56.726967 23158 raft_consensus.cc:2741] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260807 03:39:56.768523 23101 raft_consensus.cc:2781] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:39:56.786685 23122 raft_consensus.cc:3032] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 03:39:56.695878 23140 consensus_queue.cc:237] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:56.839926 18027 leader_election.cc:302] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:56.784324 22427 raft_consensus.cc:504] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:56.863616 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 499147us
I20260807 03:39:56.865068 18248 leader_election.cc:302] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 5c9b5062bdfa46af9396f1a85025a331
W20260807 03:39:56.836879 22888 consensus_meta.cc:287] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.666s user 0.007s sys 0.003s
I20260807 03:39:56.829744 18249 leader_election.cc:302] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:56.861822 18027 leader_election.cc:302] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
W20260807 03:39:56.879293 18081 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.569s user 0.007s sys 0.020s
W20260807 03:39:56.880030 18081 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.569s user 0.007s sys 0.020s
I20260807 03:39:56.841415 23101 raft_consensus.cc:482] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:56.882179 18311 raft_consensus.cc:2454] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 7.
I20260807 03:39:56.890331 18248 leader_election.cc:302] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:56.848371 23122 raft_consensus.cc:729] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5c9b5062bdfa46af9396f1a85025a331, State: Running, Role: LEADER
I20260807 03:39:56.916893 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:56.921968 18193 raft_consensus.cc:2454] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 9.
I20260807 03:39:56.922591 18297 raft_consensus.cc:2454] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 6.
I20260807 03:39:56.863943 23166 raft_consensus.cc:2781] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:39:56.921620 18248 leader_election.cc:302] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
W20260807 03:39:56.924176 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 94520.8us
I20260807 03:39:56.933694 18080 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01"
dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
mode: GRACEFUL
new_leader_uuid: "66a36e1ac8e945e893668e282d394c49"
from {username='slave'} at 127.0.0.1:55742
I20260807 03:39:56.934337 18080 raft_consensus.cc:595] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Received request to transfer leadership to TS 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:56.873066 22921 raft_consensus.cc:482] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:56.934739 18080 raft_consensus.cc:603] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260807 03:39:56.935310 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 187308us
I20260807 03:39:56.935495 18027 leader_election.cc:302] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
W20260807 03:39:56.930488 18311 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57268 (request call id 2116) took 2107 ms (client timeout 1836 ms). Trace:
W20260807 03:39:56.938161 18311 rpcz_store.cc:269] 0807 03:39:54.822817 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:54.990513 (+167696us) service_pool.cc:227] Handling call
0807 03:39:56.930290 (+1939777us) spinlock_profiling.cc:243] Waited 1.68 s on lock 0x7b54000d0554. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:56.930413 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3855175552}
I20260807 03:39:56.908519 23150 raft_consensus.cc:482] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:56.915730 23101 raft_consensus.cc:3037] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:56.887769 23124 leader_election.cc:288] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:56.941058 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:56.907955 23001 raft_consensus.cc:482] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:56.847883 23015 consensus_queue.cc:260] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:56.932858 19287 raft_consensus.cc:1226] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331 [term 8 LEADER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 7. Current term is 8. Ops: []
I20260807 03:39:56.918838 23214 raft_consensus.cc:2781] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:56.932844 23166 raft_consensus.cc:482] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:56.932991 22984 raft_consensus.cc:2741] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260807 03:39:56.935559 22426 raft_consensus.cc:2781] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader election won for term 9
I20260807 03:39:57.001621 22426 raft_consensus.cc:686] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 9 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:57.002806 22426 consensus_queue.cc:237] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:56.966406 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:57.004731 18311 raft_consensus.cc:2354] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 9. Current term is 10.
I20260807 03:39:56.974335 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:56.840348 22985 consensus_queue.cc:260] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:57.008258 22985 raft_consensus.cc:3037] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:56.952773 22932 raft_consensus.cc:2726] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 03:39:56.995442 19297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331"
I20260807 03:39:56.990671 23166 raft_consensus.cc:3037] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:56.959156 23158 raft_consensus.cc:482] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
W20260807 03:39:56.980752 22929 consensus_queue.cc:1159] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "3c4bceb836db4536a0f255fdf86abc02" responder_term: 8 status { last_received { term: 8 index: 5 } last_committed_idx: 4 last_received_current_leader { term: 8 index: 5 } }
I20260807 03:39:57.013638 23158 raft_consensus.cc:504] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:56.956585 23207 raft_consensus.cc:2781] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:39:57.006938 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:57.007179 18193 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8c98c1d422654739af06653a512f7499"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
mode: GRACEFUL
new_leader_uuid: "66a36e1ac8e945e893668e282d394c49"
from {username='slave'} at 127.0.0.1:34758
I20260807 03:39:56.980784 23214 raft_consensus.cc:482] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 5c9b5062bdfa46af9396f1a85025a331)
I20260807 03:39:57.006121 18027 leader_election.cc:398] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 pre-election: Vote denied by peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021) with higher term. Message: Invalid argument: T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 9. Current term is 10.
I20260807 03:39:57.015841 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:57.066238 18193 raft_consensus.cc:595] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Received request to transfer leadership to TS 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:57.077105 18193 raft_consensus.cc:603] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 03:39:56.957330 22921 raft_consensus.cc:504] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:56.990968 23116 leader_election.cc:288] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Requested vote from peers 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:56.958781 23150 raft_consensus.cc:504] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:57.108249 16154 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260807 03:39:57.038488 23015 raft_consensus.cc:3037] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:57.070328 22427 leader_election.cc:288] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:57.037276 23207 raft_consensus.cc:482] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:57.108870 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:56.964407 23001 raft_consensus.cc:504] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:57.095986 22993 raft_consensus.cc:482] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:57.062912 23214 raft_consensus.cc:3037] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:57.115410 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:57.115504 18311 raft_consensus.cc:2379] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 6: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:56.981717 23122 consensus_queue.cc:260] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:57.115501 23207 raft_consensus.cc:3037] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:57.118000 22993 raft_consensus.cc:504] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:57.123917 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:57.126986 18310 raft_consensus.cc:2454] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 8.
I20260807 03:39:57.128965 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:57.132584 18310 raft_consensus.cc:2454] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 7.
I20260807 03:39:57.005971 23206 consensus_queue.cc:1035] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.144s
W20260807 03:39:57.153411 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 391214us
I20260807 03:39:57.168241 22427 raft_consensus.cc:482] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:57.177974 18140 leader_election.cc:302] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:57.119078 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
W20260807 03:39:57.185169 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 179176us
I20260807 03:39:57.182780 22427 raft_consensus.cc:504] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:57.189757 18075 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33254 (request call id 2400) took 1936 ms. Trace:
I20260807 03:39:57.190806 16154 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 03:39:57.190268 18075 rpcz_store.cc:276] 0807 03:39:55.265819 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:56.507619 (+1241800us) service_pool.cc:227] Handling call
0807 03:39:56.545667 (+ 38048us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:56.545949 (+ 282us) raft_consensus.cc:1520] Early marking committed up to index 6
0807 03:39:56.926042 (+380093us) log.cc:842] Serialized 14 byte log entry
0807 03:39:57.189455 (+263413us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:57.189513 (+ 58us) raft_consensus.cc:1631] Marking committed up to 6
0807 03:39:57.189561 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:57.189618 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:57.189710 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":466641,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":40,"thread_start_us":554431,"threads_started":2,"wal-append.queue_time_us":192512}
I20260807 03:39:57.191681 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260807 03:39:57.209378 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 268877us
W20260807 03:39:57.215371 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 276097us
I20260807 03:39:57.217145 18141 leader_election.cc:302] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:57.219972 18312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57276 (request call id 1769) took 1006 ms. Trace:
I20260807 03:39:57.188472 22441 raft_consensus.cc:482] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:57.188287 23229 raft_consensus.cc:2781] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260807 03:39:57.245396 18311 raft_consensus.cc:3037] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
W20260807 03:39:57.221786 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 265223us
I20260807 03:39:57.252275 18249 leader_election.cc:302] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:57.255786 18081 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33268 (request call id 2051) took 2764 ms. Trace:
I20260807 03:39:57.220520 18312 rpcz_store.cc:276] 0807 03:39:56.214015 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:56.427624 (+213609us) service_pool.cc:227] Handling call
0807 03:39:56.427884 (+ 260us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:56.860262 (+432378us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:56.860291 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:39:56.867585 (+ 7294us) log.cc:842] Serialized 26 byte log entry
0807 03:39:57.219584 (+351999us) raft_consensus.cc:1631] Marking committed up to 4
0807 03:39:57.219651 (+ 67us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:57.219731 (+ 80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:57.219771 (+ 40us) raft_consensus.cc:1671] finished
0807 03:39:57.219818 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:57.219920 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":170129,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":15280,"thread_start_us":728280,"threads_started":2,"wal-append.queue_time_us":329352}
I20260807 03:39:57.256198 18081 rpcz_store.cc:276] 0807 03:39:54.491045 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:56.295720 (+1804675us) service_pool.cc:227] Handling call
0807 03:39:56.296034 (+ 314us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:56.296370 (+ 336us) raft_consensus.cc:1520] Early marking committed up to index 5
0807 03:39:56.880533 (+584163us) log.cc:842] Serialized 14 byte log entry
0807 03:39:57.255441 (+374908us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:57.255507 (+ 66us) raft_consensus.cc:1631] Marking committed up to 5
0807 03:39:57.255559 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:57.255626 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:57.255735 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":49514,"prepare.queue_time_us":716008,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":40,"thread_start_us":894889,"threads_started":2,"wal-append.queue_time_us":325357}
I20260807 03:39:57.217298 18027 leader_election.cc:302] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:57.220856 22932 raft_consensus.cc:482] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 03:39:57.267478 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 65029.3us
W20260807 03:39:57.229877 23205 log.cc:924] Time spent T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.423s user 0.002s sys 0.000s
I20260807 03:39:57.218350 23230 raft_consensus.cc:482] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 03:39:57.274739 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 57679.7us
I20260807 03:39:57.276641 18082 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33254 (request call id 2379) took 2535 ms. Trace:
I20260807 03:39:57.283576 18140 leader_election.cc:302] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
W20260807 03:39:57.258196 22980 consensus_meta.cc:287] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 1.030s user 0.018s sys 0.005s
W20260807 03:39:57.258045 22862 consensus_meta.cc:287] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 0.591s user 0.016s sys 0.006s
I20260807 03:39:57.223451 23229 raft_consensus.cc:482] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:57.288720 18082 rpcz_store.cc:276] 0807 03:39:54.740838 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:56.352023 (+1611185us) service_pool.cc:227] Handling call
0807 03:39:56.352356 (+ 333us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:56.359208 (+ 6852us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:56.359234 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:39:56.451235 (+ 92001us) log.cc:842] Serialized 26 byte log entry
0807 03:39:56.889063 (+437828us) raft_consensus.cc:1631] Marking committed up to 5
0807 03:39:57.063762 (+174699us) log.cc:842] Serialized 14 byte log entry
0807 03:39:57.276212 (+212450us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:57.276307 (+ 95us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:57.276404 (+ 97us) raft_consensus.cc:1671] finished
0807 03:39:57.276422 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:57.276571 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":139856,"prepare.queue_time_us":123997,"prepare.run_cpu_time_us":25,"prepare.run_wall_time_us":26,"raft.queue_time_us":56597,"raft.run_cpu_time_us":288,"raft.run_wall_time_us":92730,"thread_start_us":655870,"threads_started":3,"wal-append.queue_time_us":426899}
I20260807 03:39:57.225517 23158 leader_election.cc:288] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:57.226285 22993 leader_election.cc:288] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:57.236858 23150 leader_election.cc:288] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:57.287717 18297 raft_consensus.cc:2454] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 8.
I20260807 03:39:57.273664 22917 raft_consensus.cc:2781] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:39:57.268651 23248 raft_consensus.cc:2726] T a21f43f2cbf24aeca08087d9e2fb9b2a P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 03:39:57.237109 22881 raft_consensus.cc:504] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:57.279073 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 61840.8us
I20260807 03:39:57.298408 18311 raft_consensus.cc:2454] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 5.
I20260807 03:39:57.220285 22896 consensus_queue.cc:1035] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.068s
I20260807 03:39:57.289435 22985 consensus_queue.cc:1035] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.015s
I20260807 03:39:57.288378 23230 raft_consensus.cc:504] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:57.304459 23150 raft_consensus.cc:2781] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 03:39:57.293322 18249 leader_election.cc:302] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:57.233418 22476 raft_consensus.cc:504] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:57.254561 22441 raft_consensus.cc:504] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:57.283669 22932 raft_consensus.cc:504] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:57.286301 23116 raft_consensus.cc:2781] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260807 03:39:57.352164 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 58902.3us
I20260807 03:39:57.353070 18249 leader_election.cc:302] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:57.321090 18140 leader_election.cc:302] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:57.337684 18027 leader_election.cc:302] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:57.354856 23041 raft_consensus.cc:2781] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 03:39:57.355370 23041 raft_consensus.cc:482] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 03:39:57.319859 18246 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.132:41381, user_credentials={real_user=slave}} blocked reactor thread for 46831.8us
I20260807 03:39:57.355777 23041 raft_consensus.cc:3037] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:57.274904 22762 raft_consensus.cc:504] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:57.357425 22762 leader_election.cc:288] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:57.357928 18300 raft_consensus.cc:1226] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:57.358999 18300 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57276 (request call id 1760) took 1260 ms. Trace:
I20260807 03:39:57.359300 18300 rpcz_store.cc:276] 0807 03:39:56.098437 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:56.359251 (+260814us) service_pool.cc:227] Handling call
0807 03:39:56.359505 (+ 254us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:57.358300 (+998795us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:57.358785 (+ 485us) spinlock_profiling.cc:243] Waited 998 ms on lock 0x7b54002b43d4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:39:57.358946 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2295757568}
I20260807 03:39:57.360126 18027 leader_election.cc:302] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:57.360461 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:57.361140 18300 raft_consensus.cc:2454] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 7.
I20260807 03:39:57.362059 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:57.362201 18141 leader_election.cc:302] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:57.363137 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:57.363698 18300 raft_consensus.cc:2454] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 6.
I20260807 03:39:57.274940 22865 raft_consensus.cc:504] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:57.365726 22865 leader_election.cc:288] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 10 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:57.294406 22980 raft_consensus.cc:2726] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057) with higher term. Message: Invalid argument: T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 5. Current term is 6.
I20260807 03:39:57.366672 18295 raft_consensus.cc:2454] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 6.
I20260807 03:39:57.368585 18295 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57276 (request call id 1776) took 1021 ms. Trace:
I20260807 03:39:57.368968 18295 rpcz_store.cc:276] 0807 03:39:56.347322 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:56.350918 (+ 3596us) service_pool.cc:227] Handling call
0807 03:39:57.368400 (+1017482us) spinlock_profiling.cc:243] Waited 940 ms on lock 0x7b540035aa54. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:57.368521 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2161110784}
I20260807 03:39:57.386911 19287 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59906 (request call id 230) took 1036 ms. Trace:
I20260807 03:39:57.387430 22917 raft_consensus.cc:482] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:57.387920 23013 raft_consensus.cc:2781] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader election won for term 8
I20260807 03:39:57.388463 23013 raft_consensus.cc:686] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:57.389353 23013 consensus_queue.cc:237] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:57.390762 23166 raft_consensus.cc:504] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:57.392778 23166 leader_election.cc:288] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:57.397317 23017 consensus_meta.cc:287] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 0.615s user 0.012s sys 0.008s
W20260807 03:39:57.339799 23134 consensus_meta.cc:287] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 1.080s user 0.023s sys 0.003s
I20260807 03:39:57.392969 22441 leader_election.cc:288] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:57.402920 23122 raft_consensus.cc:3037] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:57.398469 18298 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57276 (request call id 1737) took 3939 ms. Trace:
W20260807 03:39:57.339883 22882 consensus_meta.cc:287] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 1.305s user 0.010s sys 0.018s
W20260807 03:39:57.510808 19239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 99128.6us
I20260807 03:39:57.407790 18071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33268 (request call id 2059) took 1351 ms. Trace:
I20260807 03:39:57.294862 23229 raft_consensus.cc:3037] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Advancing to term 10
I20260807 03:39:57.454778 22980 raft_consensus.cc:2781] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260807 03:39:57.457021 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 104052us
I20260807 03:39:57.470696 18298 rpcz_store.cc:276] 0807 03:39:53.458821 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:53.459151 (+ 330us) service_pool.cc:227] Handling call
0807 03:39:55.138014 (+1678863us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:55.138426 (+ 412us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:55.138459 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:39:55.727310 (+588851us) spinlock_profiling.cc:243] Waited 38.4 ms on lock 0x7b2c00108448. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe990ba2e11 00007fe990bb2442 00007fe9921fac11 00007fe9921bf58a 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772
0807 03:39:55.957585 (+230275us) log.cc:842] Serialized 26 byte log entry
0807 03:39:56.771864 (+814279us) raft_consensus.cc:1631] Marking committed up to 4
0807 03:39:56.771928 (+ 64us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:56.772018 (+ 90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:57.015379 (+243361us) raft_consensus.cc:1671] finished
0807 03:39:57.015399 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:57.398284 (+382885us) spinlock_profiling.cc:243] Waited 1.68 s on lock 0x7b54001d1d50. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bea09 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:39:57.398393 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1184,"raft.queue_time_us":1074,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":119,"spinlock_wait_cycles":3948606464,"thread_start_us":677308,"threads_started":1,"wal-append.queue_time_us":898541}
I20260807 03:39:57.457953 23150 raft_consensus.cc:686] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 6 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:57.512080 18071 rpcz_store.cc:276] 0807 03:39:56.056447 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:56.574125 (+517678us) service_pool.cc:227] Handling call
0807 03:39:56.574583 (+ 458us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:56.703107 (+128524us) raft_consensus.cc:1520] Early marking committed up to index 6
0807 03:39:56.920789 (+217682us) log.cc:842] Serialized 14 byte log entry
0807 03:39:57.221562 (+300773us) log.cc:842] Serialized 14 byte log entry
0807 03:39:57.257533 (+ 35971us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:39:57.257601 (+ 68us) raft_consensus.cc:1631] Marking committed up to 6
0807 03:39:57.257672 (+ 71us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:57.407581 (+149909us) spinlock_profiling.cc:243] Waited 128 ms on lock 0x7b54003782d4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:39:57.407634 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:57.407738 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":49032,"prepare.queue_time_us":312823,"prepare.run_cpu_time_us":24,"prepare.run_wall_time_us":24,"raft.queue_time_us":65861,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":17859,"spinlock_wait_cycles":357303552,"thread_start_us":453974,"threads_started":2,"wal-append.queue_time_us":286994}
I20260807 03:39:57.518412 18249 leader_election.cc:302] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:39:57.264259 23243 consensus_queue.cc:1035] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.187s
I20260807 03:39:57.300024 22862 raft_consensus.cc:504] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:57.387920 22917 raft_consensus.cc:3037] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Advancing to term 7
W20260807 03:39:57.537340 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 175246us
I20260807 03:39:57.538306 18249 leader_election.cc:302] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:39:57.408290 23116 raft_consensus.cc:482] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:57.312965 22658 raft_consensus.cc:504] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:57.540824 18072 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33254 (request call id 2382) took 2795 ms. Trace:
I20260807 03:39:57.348459 22916 consensus_queue.cc:1035] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.183s
W20260807 03:39:57.525004 23021 consensus_meta.cc:287] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.855s user 0.012s sys 0.017s
W20260807 03:39:57.535615 22931 consensus_meta.cc:287] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.775s user 0.010s sys 0.010s
I20260807 03:39:57.542786 18072 rpcz_store.cc:276] 0807 03:39:54.745716 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:56.352566 (+1606850us) service_pool.cc:227] Handling call
0807 03:39:56.353011 (+ 445us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:56.353302 (+ 291us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:56.354883 (+ 1581us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:39:56.417910 (+ 63027us) log.cc:842] Serialized 513 byte log entry
0807 03:39:56.724551 (+306641us) raft_consensus.cc:1631] Marking committed up to 5
0807 03:39:56.948604 (+224053us) log.cc:842] Serialized 14 byte log entry
0807 03:39:56.948840 (+ 236us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:56.948927 (+ 87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:57.229151 (+280224us) spinlock_profiling.cc:243] Waited 58.5 ms on lock 0x7b44000232cc. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f1a9a95 00007fe98f1aa4b2 00007fe991e52511 00007fe991e4de1a 00007fe991e41bee 00007fe991e40f36 00007fe991e40b58 00007fe991e4a551 00007fe991e4a509 00007fe991e4a499
0807 03:39:57.537183 (+308032us) raft_consensus.cc:1671] finished
0807 03:39:57.540603 (+ 3420us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:57.540766 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3692,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":105,"spinlock_wait_cycles":134516096,"thread_start_us":305818,"threads_started":1,"wal-append.queue_time_us":388824}
I20260807 03:39:57.523023 22980 raft_consensus.cc:482] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:57.517313 23248 raft_consensus.cc:2741] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader election decision vote started in defunct term 8: won
I20260807 03:39:57.554862 23230 leader_election.cc:288] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:57.517076 22882 raft_consensus.cc:2726] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260807 03:39:57.387332 19287 rpcz_store.cc:276] 0807 03:39:56.350673 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:56.357722 (+ 7049us) service_pool.cc:227] Handling call
0807 03:39:56.540375 (+182653us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:56.992138 (+451763us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:57.344271 (+352133us) spinlock_profiling.cc:243] Waited 367 ms on lock 0x7b5400700b54. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:39:57.386844 (+ 42573us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":844584704}
I20260807 03:39:57.538542 23261 raft_consensus.cc:2781] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:39:57.556668 23261 raft_consensus.cc:482] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
W20260807 03:39:57.557143 22897 consensus_meta.cc:287] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 1.018s user 0.016s sys 0.005s
I20260807 03:39:57.532222 22921 leader_election.cc:288] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:57.557641 22897 raft_consensus.cc:504] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:57.557904 23150 consensus_queue.cc:237] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:57.559701 22897 leader_election.cc:288] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:57.535105 22762 raft_consensus.cc:2781] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 03:39:57.560747 22762 raft_consensus.cc:482] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:57.561182 22762 raft_consensus.cc:3037] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:57.557077 23261 raft_consensus.cc:3037] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:57.539935 23242 raft_consensus.cc:2726] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260807 03:39:57.563110 23010 consensus_meta.cc:287] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 0.774s user 0.006s sys 0.024s
I20260807 03:39:57.564510 23214 raft_consensus.cc:504] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:57.566574 23214 leader_election.cc:288] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:57.567262 22427 leader_election.cc:288] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
W20260807 03:39:57.567739 23145 consensus_meta.cc:287] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 0.781s user 0.015s sys 0.005s
I20260807 03:39:57.567982 22980 raft_consensus.cc:3037] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:57.568908 23207 raft_consensus.cc:504] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:57.570879 23207 leader_election.cc:288] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
W20260807 03:39:57.575523 23132 consensus_meta.cc:287] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.807s user 0.022s sys 0.000s
W20260807 03:39:57.576027 22530 consensus_meta.cc:287] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 1.289s user 0.016s sys 0.007s
I20260807 03:39:57.576392 22530 raft_consensus.cc:504] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:57.577127 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:57.578215 22530 leader_election.cc:288] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:57.578405 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:57.579435 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:57.582932 22847 raft_consensus.cc:2726] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 03:39:57.582253 23229 raft_consensus.cc:504] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:57.467831 23134 raft_consensus.cc:504] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:57.588362 23134 leader_election.cc:288] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:57.550345 23021 raft_consensus.cc:504] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:57.589375 22931 raft_consensus.cc:504] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:57.587013 23261 raft_consensus.cc:504] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:57.590786 23021 leader_election.cc:288] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:57.591357 22931 leader_election.cc:288] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:57.591946 18293 raft_consensus.cc:1226] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 8. Current term is 9. Ops: []
I20260807 03:39:57.593379 18305 raft_consensus.cc:1226] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 4. Current term is 5. Ops: []
I20260807 03:39:57.593446 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:57.593542 23229 leader_election.cc:288] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:57.597806 18293 raft_consensus.cc:2454] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 8.
I20260807 03:39:57.630242 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:57.630383 18141 leader_election.cc:302] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:57.631183 18293 raft_consensus.cc:2379] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 10: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:57.598096 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:57.631747 18305 raft_consensus.cc:3037] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:57.519348 22441 raft_consensus.cc:2726] T d24d0bbea8bd49d482ad95538ca6e631 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260807 03:39:57.547428 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 242079us
I20260807 03:39:57.604265 19290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:57.604988 19287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331"
I20260807 03:39:57.666440 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:57.667315 18080 raft_consensus.cc:2396] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260807 03:39:57.606499 22881 leader_election.cc:288] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:57.609401 18003 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 303e3bbfb1c74e1686eae36683a96314, TS 66a36e1ac8e945e893668e282d394c49 -> TS 5c9b5062bdfa46af9396f1a85025a331 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:57.614921 23134 consensus_queue.cc:1046] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 03:39:57.615478 22476 leader_election.cc:288] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:57.615674 22862 leader_election.cc:288] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:57.616282 23201 consensus_queue.cc:1046] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 03:39:57.627965 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:57.635301 19289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:57.482271 23017 raft_consensus.cc:504] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:57.650723 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 290702us
I20260807 03:39:57.687352 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:57.652140 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:57.545224 23116 raft_consensus.cc:3037] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:57.516273 23101 raft_consensus.cc:504] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:57.710258 23101 leader_election.cc:288] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:57.712394 23201 raft_consensus.cc:3032] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 03:39:57.712877 23201 raft_consensus.cc:729] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:57.713730 22917 raft_consensus.cc:504] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:57.582686 23137 consensus_meta.cc:287] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 0.824s user 0.023s sys 0.005s
I20260807 03:39:57.664336 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:57.715387 23247 raft_consensus.cc:482] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:57.715583 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:57.668612 18082 raft_consensus.cc:1261] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260807 03:39:57.716935 18081 raft_consensus.cc:1226] T 4d3a25995c8341c0bf39232774e3a917 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 7. Current term is 8. Ops: []
I20260807 03:39:57.522641 23041 raft_consensus.cc:504] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:57.669777 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:57.717918 18080 raft_consensus.cc:3037] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Advancing to term 10
I20260807 03:39:57.671794 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:57.676774 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:57.718897 18070 raft_consensus.cc:2379] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 11: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:57.680227 18305 raft_consensus.cc:2454] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 7.
W20260807 03:39:57.669346 18003 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 303e3bbfb1c74e1686eae36683a96314, TS 66a36e1ac8e945e893668e282d394c49 -> TS 5c9b5062bdfa46af9396f1a85025a331 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:57.685137 18297 raft_consensus.cc:1226] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 6. Current term is 7. Ops: []
I20260807 03:39:57.677635 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:57.719658 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:57.575932 23132 raft_consensus.cc:504] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:57.722404 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:57.722936 23132 leader_election.cc:288] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:57.723218 19289 raft_consensus.cc:2454] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 8.
I20260807 03:39:57.723455 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:57.723637 18300 raft_consensus.cc:1226] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 6. Current term is 7. Ops: []
I20260807 03:39:57.724454 19297 raft_consensus.cc:2396] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260807 03:39:57.668279 23229 raft_consensus.cc:2781] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 03:39:57.693323 22658 leader_election.cc:288] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:57.662824 22984 consensus_queue.cc:1046] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.042s
W20260807 03:39:57.695758 22902 consensus_meta.cc:287] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 0.965s user 0.018s sys 0.011s
I20260807 03:39:57.696360 18325 tablet_copy_service.cc:140] P 66a36e1ac8e945e893668e282d394c49: Received BeginTabletCopySession request for tablet d416e1015a2644958b0c8f9147d403a2 from peer 5c9b5062bdfa46af9396f1a85025a331 ({username='slave'} at 127.0.0.1:37190)
I20260807 03:39:57.696539 23261 leader_election.cc:288] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:57.697059 23109 consensus_meta.cc:287] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 1.016s user 0.012s sys 0.013s
I20260807 03:39:57.696919 22862 consensus_queue.cc:1046] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 03:39:57.563496 23010 raft_consensus.cc:504] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:57.759202 18325 tablet_copy_service.cc:161] P 66a36e1ac8e945e893668e282d394c49: Beginning new tablet copy session on tablet d416e1015a2644958b0c8f9147d403a2 from peer 5c9b5062bdfa46af9396f1a85025a331 at {username='slave'} at 127.0.0.1:37190: session id = 5c9b5062bdfa46af9396f1a85025a331-d416e1015a2644958b0c8f9147d403a2
I20260807 03:39:57.759963 22932 leader_election.cc:288] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:57.601523 23145 raft_consensus.cc:504] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:57.760530 23010 leader_election.cc:288] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:57.680939 23040 consensus_meta.cc:287] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 0.903s user 0.012s sys 0.012s
I20260807 03:39:57.761245 23040 raft_consensus.cc:504] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:57.761898 23145 leader_election.cc:288] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 election: Requested vote from peers 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:57.700214 23017 leader_election.cc:288] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 11 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:57.763326 23040 leader_election.cc:288] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 12 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:57.763924 18190 raft_consensus.cc:1226] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 12 FOLLOWER]: Rejecting Update request from peer 5c9b5062bdfa46af9396f1a85025a331 for earlier term 11. Current term is 12. Ops: []
I20260807 03:39:57.703018 22881 raft_consensus.cc:3032] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 8 LEADER]: Stepping down as leader of term 8
I20260807 03:39:57.701100 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:57.764608 22881 raft_consensus.cc:729] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:57.764726 18138 leader_election.cc:302] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:57.704363 18298 raft_consensus.cc:2379] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 9: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:57.691570 18027 leader_election.cc:302] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:57.765478 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:57.732178 22917 leader_election.cc:288] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:57.713696 23201 consensus_queue.cc:260] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:57.718807 18071 raft_consensus.cc:2454] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 6.
I20260807 03:39:57.598212 23001 leader_election.cc:288] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:57.687623 18075 raft_consensus.cc:1226] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 7. Current term is 8. Ops: []
I20260807 03:39:57.720261 18305 raft_consensus.cc:3037] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:57.719658 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
W20260807 03:39:57.773383 18138 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.132:41381, user_credentials={real_user=slave}} blocked reactor thread for 46055.9us
I20260807 03:39:57.773931 18070 raft_consensus.cc:3037] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:57.671133 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:57.715885 23247 raft_consensus.cc:504] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:57.721221 18141 leader_election.cc:302] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:57.735723 22762 raft_consensus.cc:504] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:57.781986 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 60844.6us
I20260807 03:39:57.782224 23017 raft_consensus.cc:3032] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:39:57.782711 23017 raft_consensus.cc:729] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:57.783149 22762 leader_election.cc:288] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:57.784407 22658 raft_consensus.cc:2781] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 03:39:57.784896 22658 raft_consensus.cc:686] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:57.785342 23041 leader_election.cc:288] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:57.785713 22658 consensus_queue.cc:237] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:57.724722 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ca0223f47b46fd955704eba5259370" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:57.788524 18082 raft_consensus.cc:2454] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:39:57.788165 22881 consensus_queue.cc:260] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:57.789393 22881 raft_consensus.cc:3037] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:57.789633 23247 leader_election.cc:288] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:57.790500 23270 consensus_queue.cc:1046] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 03:39:57.791569 22897 raft_consensus.cc:482] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:57.791935 23270 raft_consensus.cc:482] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:57.792143 22896 raft_consensus.cc:482] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:57.792119 22897 raft_consensus.cc:504] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:57.792544 23270 raft_consensus.cc:504] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:57.792845 22896 raft_consensus.cc:504] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:39:57.794134 22897 leader_election.cc:288] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:57.794790 22897 raft_consensus.cc:3032] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Stepping down as leader of term 6
I20260807 03:39:57.795039 22896 leader_election.cc:288] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:57.795087 23270 leader_election.cc:288] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:57.783629 23017 consensus_queue.cc:260] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:57.797237 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:57.797592 23017 raft_consensus.cc:3037] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:57.798336 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:57.799173 18297 raft_consensus.cc:2396] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5c9b5062bdfa46af9396f1a85025a331 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260807 03:39:57.795133 22897 raft_consensus.cc:729] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:57.801187 22897 consensus_queue.cc:260] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:57.802739 22897 raft_consensus.cc:3037] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:57.804382 18297 raft_consensus.cc:1226] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Rejecting Update request from peer 5c9b5062bdfa46af9396f1a85025a331 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20260807 03:39:57.661753 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 340719us
I20260807 03:39:57.734369 18081 raft_consensus.cc:2379] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 5: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:57.725824 18310 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57268 (request call id 2137) took 1062 ms. Trace:
I20260807 03:39:57.707561 22980 raft_consensus.cc:504] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:57.819159 18293 raft_consensus.cc:2379] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 9: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:57.819612 18325 tablet_copy_source_session.cc:205] T d416e1015a2644958b0c8f9147d403a2 P 66a36e1ac8e945e893668e282d394c49: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 03:39:57.819866 22980 leader_election.cc:288] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:57.808619 18310 rpcz_store.cc:276] 0807 03:39:56.663106 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:57.203510 (+540404us) service_pool.cc:227] Handling call
0807 03:39:57.254844 (+ 51334us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:57.260855 (+ 6011us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:57.260899 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:39:57.286448 (+ 25549us) log.cc:842] Serialized 26 byte log entry
0807 03:39:57.349493 (+ 63045us) raft_consensus.cc:1631] Marking committed up to 5
0807 03:39:57.725216 (+375723us) log.cc:842] Serialized 14 byte log entry
0807 03:39:57.725475 (+ 259us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:57.725568 (+ 93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:57.725609 (+ 41us) raft_consensus.cc:1671] finished
0807 03:39:57.725625 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:57.725759 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10883,"prepare.queue_time_us":422046,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":32,"raft.queue_time_us":26417,"raft.run_cpu_time_us":291,"raft.run_wall_time_us":16966,"thread_start_us":248639,"threads_started":2,"wal-append.queue_time_us":406777}
I20260807 03:39:57.821234 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:57.821743 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:57.822571 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:57.825526 18310 raft_consensus.cc:1226] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Rejecting Update request from peer 5c9b5062bdfa46af9396f1a85025a331 for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:57.752930 18073 raft_consensus.cc:2379] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 8: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:57.829598 22881 consensus_queue.cc:1046] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 03:39:57.834074 22881 raft_consensus.cc:3032] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Stepping down as leader of term 6
I20260807 03:39:57.834422 22881 raft_consensus.cc:729] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:57.754292 18069 raft_consensus.cc:1226] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260807 03:39:57.757772 18248 leader_election.cc:302] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:57.752430 22902 raft_consensus.cc:504] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:57.766032 18079 raft_consensus.cc:3037] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:57.838138 22980 raft_consensus.cc:2726] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260807 03:39:57.838752 22902 leader_election.cc:288] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 10 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:57.704954 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:57.840159 18293 raft_consensus.cc:1261] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Refusing update from remote peer 5c9b5062bdfa46af9396f1a85025a331: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260807 03:39:57.839779 22881 consensus_queue.cc:260] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:57.712328 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:57.764621 18189 raft_consensus.cc:1226] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Rejecting Update request from peer 5c9b5062bdfa46af9396f1a85025a331 for earlier term 7. Current term is 8. Ops: []
I20260807 03:39:57.735481 23229 raft_consensus.cc:482] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:57.843076 23229 raft_consensus.cc:3037] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:57.768456 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:57.775271 19297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331"
W20260807 03:39:57.765720 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 74229.8us
I20260807 03:39:57.776528 23001 raft_consensus.cc:2781] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 03:39:57.776633 23133 consensus_queue.cc:1046] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260807 03:39:57.776566 23201 raft_consensus.cc:3037] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:57.752053 23132 raft_consensus.cc:482] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:57.777860 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:57.851199 18075 raft_consensus.cc:2379] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 5: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:57.752072 18080 raft_consensus.cc:2454] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 10.
I20260807 03:39:57.765434 23109 raft_consensus.cc:504] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:57.773738 23101 raft_consensus.cc:2781] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:39:57.779539 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:57.852366 18071 raft_consensus.cc:2379] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 9: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:57.779886 18305 raft_consensus.cc:2454] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 9.
I20260807 03:39:57.779711 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:57.855610 18141 leader_election.cc:302] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:57.808610 18140 leader_election.cc:302] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:57.816116 19287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331"
I20260807 03:39:57.818243 18081 raft_consensus.cc:1261] T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Refusing update from remote peer 5c9b5062bdfa46af9396f1a85025a331: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260807 03:39:57.858688 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 319209us
I20260807 03:39:57.859367 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:57.860299 18141 leader_election.cc:302] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:57.812969 18070 raft_consensus.cc:2454] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 7.
W20260807 03:39:57.858855 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 50332.4us
I20260807 03:39:57.840531 18293 pending_rounds.cc:76] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49: Aborting all ops after (but not including) 5
I20260807 03:39:57.861476 18293 pending_rounds.cc:97] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260807 03:39:57.861609 18140 leader_election.cc:302] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:57.861831 18293 raft_consensus.cc:2866] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 03:39:57.844920 18189 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50060 (request call id 205) took 1259 ms. Trace:
I20260807 03:39:57.862792 18309 raft_consensus.cc:1226] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:57.863170 18189 rpcz_store.cc:276] 0807 03:39:56.585124 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:56.585292 (+ 168us) service_pool.cc:227] Handling call
0807 03:39:56.585637 (+ 345us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:57.843014 (+1257377us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:57.844718 (+ 1704us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b54002a2954. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:39:57.844867 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2710996736}
I20260807 03:39:57.845173 18027 leader_election.cc:302] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:57.755328 22984 consensus_queue.cc:1225] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer 5c9b5062bdfa46af9396f1a85025a331 log is divergent from this leader: its last log entry 8.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260807 03:39:57.846902 18298 raft_consensus.cc:2379] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 9: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:57.849594 18079 raft_consensus.cc:2454] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 7.
I20260807 03:39:57.847601 23001 raft_consensus.cc:482] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:57.873323 18140 leader_election.cc:302] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:57.874828 18140 leader_election.cc:302] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:57.878558 18309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57268 (request call id 2134) took 1218 ms. Trace:
I20260807 03:39:57.879113 18027 leader_election.cc:302] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:39:57.879067 18309 rpcz_store.cc:276] 0807 03:39:56.661971 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:56.759451 (+ 97480us) service_pool.cc:227] Handling call
0807 03:39:56.759771 (+ 320us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:39:57.863341 (+1103570us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:57.878277 (+ 14936us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54002b3ed4. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:39:57.878494 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2536158464}
I20260807 03:39:57.875557 23247 raft_consensus.cc:482] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:57.882231 23247 raft_consensus.cc:504] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:57.839006 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 81298.5us
I20260807 03:39:57.834968 23116 raft_consensus.cc:504] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:57.889448 18300 raft_consensus.cc:2454] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 7.
I20260807 03:39:57.892596 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:57.894913 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:57.895658 18309 raft_consensus.cc:2379] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 6: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:57.897033 18141 leader_election.cc:302] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:57.898309 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:57.898653 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:57.899063 18070 raft_consensus.cc:1226] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:57.900000 18079 raft_consensus.cc:1226] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 4. Current term is 5. Ops: []
I20260807 03:39:57.862486 23040 raft_consensus.cc:2726] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260807 03:39:57.872140 23271 raft_consensus.cc:2726] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:57.904673 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:57.885437 22441 raft_consensus.cc:2781] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 03:39:57.880499 22902 raft_consensus.cc:2741] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260807 03:39:57.908284 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:57.989233 18070 raft_consensus.cc:3037] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:57.880786 22896 raft_consensus.cc:2781] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader election won for term 9
I20260807 03:39:57.992740 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:57.917913 18300 raft_consensus.cc:2379] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 5: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:57.993384 18293 raft_consensus.cc:3037] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:57.878664 23001 raft_consensus.cc:3037] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:57.890367 22862 raft_consensus.cc:2781] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election won for term 5
I20260807 03:39:57.855775 23132 raft_consensus.cc:504] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:57.980232 18295 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf"
dest_uuid: "66a36e1ac8e945e893668e282d394c49"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57260
I20260807 03:39:58.003669 18295 raft_consensus.cc:595] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 9 LEADER]: Received request to transfer leadership
I20260807 03:39:58.004019 18070 raft_consensus.cc:2454] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 7.
I20260807 03:39:57.884493 23134 raft_consensus.cc:2781] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader election won for term 8
I20260807 03:39:57.980767 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:58.006196 18298 raft_consensus.cc:2454] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 10.
I20260807 03:39:57.981449 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:57.984895 18310 raft_consensus.cc:3032] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Stepping down as leader of term 7
I20260807 03:39:58.007021 18079 raft_consensus.cc:3037] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:58.007213 18310 raft_consensus.cc:729] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:57.986975 18081 raft_consensus.cc:1261] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260807 03:39:57.862618 23167 consensus_queue.cc:1035] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20260807 03:39:57.896713 23158 raft_consensus.cc:2781] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 03:39:57.992030 22441 raft_consensus.cc:482] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:57.996970 23247 leader_election.cc:288] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:58.023427 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:58.024163 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:58.024333 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8877c3fbba4ab18765832f1390921f" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:58.024449 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:58.024683 18073 raft_consensus.cc:3037] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:58.024848 18075 raft_consensus.cc:1226] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 6. Current term is 7. Ops: []
I20260807 03:39:58.025143 18069 raft_consensus.cc:2379] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 10: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:58.025520 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:58.025817 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:58.025935 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:58.026317 18188 raft_consensus.cc:2379] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 6: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:58.026805 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:58.027395 18183 raft_consensus.cc:2454] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 5.
I20260807 03:39:58.024366 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:58.030413 18071 raft_consensus.cc:2454] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 9.
I20260807 03:39:58.010337 22896 raft_consensus.cc:686] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:58.016373 22530 raft_consensus.cc:482] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:58.032764 22530 raft_consensus.cc:504] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:58.034749 22530 leader_election.cc:288] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:57.997534 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
W20260807 03:39:58.000488 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 130140us
I20260807 03:39:57.886241 23133 consensus_queue.cc:1046] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.103s
I20260807 03:39:58.041052 23133 raft_consensus.cc:3032] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 7 LEADER]: Stepping down as leader of term 7
I20260807 03:39:58.041378 23133 raft_consensus.cc:729] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 5c9b5062bdfa46af9396f1a85025a331, State: Running, Role: LEADER
I20260807 03:39:58.042177 23133 consensus_queue.cc:260] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:58.043254 23133 raft_consensus.cc:3037] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:57.917768 22881 raft_consensus.cc:3037] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:58.044189 18079 raft_consensus.cc:2454] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 6.
I20260807 03:39:58.045337 18070 raft_consensus.cc:1226] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 9. Current term is 10. Ops: []
I20260807 03:39:58.046238 18073 raft_consensus.cc:2454] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 7.
I20260807 03:39:58.046753 23101 raft_consensus.cc:482] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:58.047168 23101 raft_consensus.cc:3037] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:57.905542 22921 raft_consensus.cc:2781] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 03:39:58.048518 22426 consensus_queue.cc:1046] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 03:39:58.049892 22426 raft_consensus.cc:3032] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 6 LEADER]: Stepping down as leader of term 6
I20260807 03:39:58.050258 22426 raft_consensus.cc:729] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:58.051102 22426 consensus_queue.cc:260] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:58.009507 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 122442us
I20260807 03:39:58.053197 23132 leader_election.cc:288] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:58.053730 18248 leader_election.cc:302] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:58.053908 23132 consensus_queue.cc:1035] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 03:39:58.054226 23101 raft_consensus.cc:504] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:58.054910 18248 leader_election.cc:302] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:58.055770 18075 raft_consensus.cc:2454] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 9.
I20260807 03:39:58.055868 18309 raft_consensus.cc:2454] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 9.
W20260807 03:39:58.056818 23262 log.cc:924] Time spent T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20260807 03:39:58.058003 23109 leader_election.cc:288] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:57.995200 23040 raft_consensus.cc:482] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:58.060542 23040 raft_consensus.cc:504] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:57.896206 22476 consensus_queue.cc:1046] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.107s
I20260807 03:39:58.013988 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:58.015237 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:39:58.015344 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 120576us
I20260807 03:39:58.062126 23101 leader_election.cc:288] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:58.062569 23040 leader_election.cc:288] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:58.062141 18187 raft_consensus.cc:2454] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 5.
I20260807 03:39:58.065730 18027 leader_election.cc:302] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:58.066283 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:58.068261 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:58.068370 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:58.017769 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:58.018738 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:58.069006 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:58.013760 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:58.020973 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:58.023027 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:58.049703 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:58.016657 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:58.048460 23207 consensus_queue.cc:1046] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 03:39:58.076424 18027 leader_election.cc:302] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:58.069490 18180 raft_consensus.cc:2454] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 9.
I20260807 03:39:58.077032 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:58.078135 18193 raft_consensus.cc:2379] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 6: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:58.008275 18310 consensus_queue.cc:260] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:58.079792 18310 raft_consensus.cc:3037] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:58.071373 18185 raft_consensus.cc:2454] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 13.
I20260807 03:39:58.000689 18312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57268 (request call id 2138) took 1337 ms. Trace:
I20260807 03:39:58.072373 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:58.080986 18081 raft_consensus.cc:2454] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 8.
I20260807 03:39:58.081182 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:58.080766 18312 rpcz_store.cc:276] 0807 03:39:56.680599 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:57.259840 (+579241us) service_pool.cc:227] Handling call
0807 03:39:57.260275 (+ 435us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 03:39:57.261390 (+ 1115us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:57.261430 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 03:39:57.275901 (+ 14471us) log.cc:842] Serialized 52 byte log entry
0807 03:39:57.391994 (+116093us) spinlock_profiling.cc:243] Waited 5.09 ms on lock 0x7b48002802d8. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe9970667f1 00007fe99705d8e7 00007fe99705d77a 00007fe99715195f 00007fe997151627 00007fe997165b2c 00007fe997165ad8 00007fe997165a58 00007fe997165a13 00007fe997164cbc 00007fe9960e6be6
0807 03:39:57.519937 (+127943us) raft_consensus.cc:1631] Marking committed up to 6
0807 03:39:57.858270 (+338333us) log.cc:842] Serialized 14 byte log entry
0807 03:39:57.859358 (+ 1088us) log.cc:842] Serialized 14 byte log entry
0807 03:39:58.000215 (+140857us) spinlock_profiling.cc:243] Waited 2.37 ms on lock 0x7b18001d6f20. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe9981368b1 00007fe998143661 00007fe98f20e4ca 00007fe9988eff99 00007fe98f0d2990 00007fe98f294864 00007fe98f294155 00007fe9921b73d0 00007fe9921c9576 00007fe9921da0d7 00007fe9921da058
0807 03:39:58.000405 (+ 190us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:58.000472 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:58.000510 (+ 38us) raft_consensus.cc:1671] finished
0807 03:39:58.000521 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:58.000637 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":229562,"prepare.queue_time_us":308124,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":84,"raft.queue_time_us":152839,"raft.run_cpu_time_us":2962,"raft.run_wall_time_us":132207,"spinlock_wait_cycles":17166592,"thread_start_us":535500,"threads_started":2,"wal-append.queue_time_us":331148}
I20260807 03:39:58.083930 18027 leader_election.cc:302] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:58.085940 18027 leader_election.cc:302] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:58.093799 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:58.094560 18180 raft_consensus.cc:3037] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:58.095273 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:58.095839 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:58.098898 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:58.101905 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:58.102228 18183 raft_consensus.cc:1226] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 9. Current term is 10. Ops: []
I20260807 03:39:58.067414 22441 raft_consensus.cc:3037] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:58.138198 18027 leader_election.cc:302] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
W20260807 03:39:58.134400 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 60171.3us
I20260807 03:39:58.140151 18248 leader_election.cc:302] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:58.133695 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:58.076674 22932 raft_consensus.cc:2781] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260807 03:39:58.150360 18193 raft_consensus.cc:2454] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 10.
I20260807 03:39:58.148296 18293 raft_consensus.cc:2454] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 8.
I20260807 03:39:58.015698 22862 raft_consensus.cc:686] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:58.152292 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:58.152391 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:58.084311 22984 raft_consensus.cc:2741] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260807 03:39:58.031683 22896 consensus_queue.cc:237] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:58.093904 22902 raft_consensus.cc:2726] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260807 03:39:58.185818 22902 raft_consensus.cc:482] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:58.186383 22902 raft_consensus.cc:504] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:58.188123 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:58.188246 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:58.188369 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:58.188915 18070 raft_consensus.cc:2354] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 7. Current term is 8.
I20260807 03:39:58.189038 18188 raft_consensus.cc:2454] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 4.
I20260807 03:39:58.190017 18079 raft_consensus.cc:1226] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 11 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260807 03:39:58.189975 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:58.190639 18188 raft_consensus.cc:3037] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:58.191537 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbf00d2d944c18921cc32b7c48be16" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:58.193225 22909 raft_consensus.cc:3032] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [term 11 LEADER]: Stepping down as leader of term 11
I20260807 03:39:58.193624 22909 raft_consensus.cc:729] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 5c9b5062bdfa46af9396f1a85025a331, State: Running, Role: LEADER
I20260807 03:39:58.194209 22862 consensus_queue.cc:237] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:58.194499 22909 consensus_queue.cc:260] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:58.194960 23229 raft_consensus.cc:504] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:58.196352 18190 raft_consensus.cc:2354] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 8. Current term is 9.
I20260807 03:39:58.197348 18082 raft_consensus.cc:1226] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 8. Current term is 9. Ops: []
I20260807 03:39:58.197598 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:58.195662 22909 raft_consensus.cc:3037] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [term 11 FOLLOWER]: Advancing to term 12
I20260807 03:39:58.198520 18190 raft_consensus.cc:2379] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 10: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:58.199290 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:58.149777 22847 consensus_queue.cc:1046] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.048s
W20260807 03:39:58.083755 23150 consensus_queue.cc:1159] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" responder_term: 8 status { last_received { term: 8 index: 9 } last_committed_idx: 8 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 7. Current term is 8. Ops: []" } } last_received_current_leader { term: 8 index: 9 } }
I20260807 03:39:58.201093 18295 raft_consensus.cc:3037] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:58.202395 23248 raft_consensus.cc:482] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:58.202924 23248 raft_consensus.cc:504] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:58.098320 23134 raft_consensus.cc:686] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 8 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:58.188476 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:58.199822 18190 raft_consensus.cc:2454] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 7.
I20260807 03:39:58.206135 18080 raft_consensus.cc:3037] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:58.206300 18185 raft_consensus.cc:2454] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 7.
I20260807 03:39:58.206787 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb66518c65e4495e881ce860253d0801" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:58.208031 18190 raft_consensus.cc:2379] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 6: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:58.207924 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:58.100242 23013 raft_consensus.cc:2726] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260807 03:39:58.141340 18027 leader_election.cc:302] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:58.209460 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:58.084201 22980 raft_consensus.cc:2726] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 03:39:58.204820 23134 consensus_queue.cc:237] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:58.185003 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:58.226013 18080 raft_consensus.cc:2454] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 9.
I20260807 03:39:58.229470 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:39:58.230654 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 89403.8us
I20260807 03:39:58.098817 23261 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:58.098955 23137 raft_consensus.cc:2781] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260807 03:39:58.151010 23101 raft_consensus.cc:2726] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260807 03:39:58.241422 18027 leader_election.cc:302] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:39:58.204840 23248 leader_election.cc:288] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:58.140642 22921 raft_consensus.cc:686] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:58.152756 22426 raft_consensus.cc:3037] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:58.243896 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:58.277124 18193 raft_consensus.cc:3037] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Advancing to term 10
I20260807 03:39:58.255939 18027 leader_election.cc:302] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:39:58.244628 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:58.278949 18073 raft_consensus.cc:2454] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 6.
I20260807 03:39:58.132928 23158 raft_consensus.cc:686] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:58.197512 23116 leader_election.cc:288] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:58.246459 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:58.247180 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:58.293691 18080 raft_consensus.cc:2454] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 5.
I20260807 03:39:58.247746 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:58.295466 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:58.296314 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:58.297237 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:58.297659 18190 raft_consensus.cc:3037] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:58.300282 18295 raft_consensus.cc:2454] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 8.
I20260807 03:39:58.175997 18180 raft_consensus.cc:2454] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 6.
I20260807 03:39:58.226624 22932 raft_consensus.cc:482] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 03:39:58.234368 18003 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 922e1d3cde8f476ea7780363c6b8693c, TS d9716e2bf06442f5b3b0e81087c53c4d -> TS 5c9b5062bdfa46af9396f1a85025a331 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:58.252928 23137 raft_consensus.cc:482] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:58.259156 22993 consensus_queue.cc:1035] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 03:39:58.259617 22902 leader_election.cc:288] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:58.259806 23270 raft_consensus.cc:482] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
W20260807 03:39:58.264338 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 199393us
I20260807 03:39:58.265635 23001 raft_consensus.cc:504] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:58.268464 23264 consensus_queue.cc:1046] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 03:39:58.269157 22984 raft_consensus.cc:2781] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:39:58.280884 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:58.282198 18027 leader_election.cc:302] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:58.332541 18293 raft_consensus.cc:2379] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 4: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
W20260807 03:39:58.332892 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 53536.7us
I20260807 03:39:58.333673 18027 leader_election.cc:302] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:58.301764 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:58.337023 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:58.337829 18180 raft_consensus.cc:2379] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 9: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:58.338613 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:39:58.276674 23045 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 8 current term: 8 is higher.
I20260807 03:39:58.382520 18190 raft_consensus.cc:2454] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
W20260807 03:39:58.304725 18003 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 922e1d3cde8f476ea7780363c6b8693c, TS d9716e2bf06442f5b3b0e81087c53c4d -> TS 5c9b5062bdfa46af9396f1a85025a331 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:58.236267 22917 raft_consensus.cc:2781] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 03:39:58.306456 18193 raft_consensus.cc:2454] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 10.
I20260807 03:39:58.256991 23013 raft_consensus.cc:2726] T d24d0bbea8bd49d482ad95538ca6e631 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:58.283640 23101 raft_consensus.cc:2726] T af8877c3fbba4ab18765832f1390921f P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 03:39:58.327648 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:58.305409 23229 leader_election.cc:288] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:58.328373 18140 leader_election.cc:302] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:58.307051 22932 raft_consensus.cc:3037] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Advancing to term 11
I20260807 03:39:58.345520 18309 raft_consensus.cc:1226] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 8. Current term is 9. Ops: []
I20260807 03:39:58.347151 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:58.397246 18312 raft_consensus.cc:2379] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 9: Already voted for candidate 66a36e1ac8e945e893668e282d394c49 in this term.
I20260807 03:39:58.347800 18027 leader_election.cc:302] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:58.303915 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:58.346132 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:58.253372 23261 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:58.389154 18073 raft_consensus.cc:2379] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 10: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
W20260807 03:39:58.404732 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 57017.8us
I20260807 03:39:58.405404 18027 leader_election.cc:302] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:58.383484 23116 raft_consensus.cc:2781] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:39:58.327131 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:58.382202 22984 raft_consensus.cc:482] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:58.388239 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:58.325732 23137 raft_consensus.cc:3037] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Advancing to term 11
I20260807 03:39:58.280139 23145 consensus_queue.cc:1046] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.051s
W20260807 03:39:58.392874 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 64808.3us
I20260807 03:39:58.395002 18293 raft_consensus.cc:1226] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 8. Current term is 9. Ops: []
I20260807 03:39:58.398048 18310 raft_consensus.cc:3037] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:58.402310 23257 raft_consensus.cc:2781] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 03:39:58.409601 18312 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf"
dest_uuid: "66a36e1ac8e945e893668e282d394c49"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57260
I20260807 03:39:58.402151 23302 raft_consensus.cc:482] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:58.385245 22917 raft_consensus.cc:482] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:58.412573 18189 raft_consensus.cc:2454] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 9.
I20260807 03:39:58.332188 23270 raft_consensus.cc:504] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:58.437846 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:39:58.394297 23017 consensus_meta.cc:287] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 0.596s user 0.010s sys 0.000s
I20260807 03:39:58.423153 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:58.382817 22902 raft_consensus.cc:2726] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:58.431895 18312 raft_consensus.cc:595] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 9 LEADER]: Received request to transfer leadership
I20260807 03:39:58.388835 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
W20260807 03:39:58.429914 23201 consensus_meta.cc:287] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 0.574s user 0.007s sys 0.003s
W20260807 03:39:58.424970 23301 log.cc:924] Time spent T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.149s user 0.000s sys 0.001s
I20260807 03:39:58.439814 18190 raft_consensus.cc:2379] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 9: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:58.440701 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:58.442183 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ca0223f47b46fd955704eba5259370" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:58.327235 22921 consensus_queue.cc:237] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:58.438629 22476 consensus_queue.cc:1035] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20260807 03:39:58.442833 22862 raft_consensus.cc:2781] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader election won for term 8
I20260807 03:39:58.443022 22917 raft_consensus.cc:3037] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:58.413374 23116 raft_consensus.cc:482] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:58.473219 23116 raft_consensus.cc:3037] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:58.480651 18185 raft_consensus.cc:1226] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:58.484122 22847 raft_consensus.cc:979] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49: : Instructing follower 5c9b5062bdfa46af9396f1a85025a331 to start an election
I20260807 03:39:58.484782 22847 raft_consensus.cc:1067] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 9 LEADER]: Signalling peer 5c9b5062bdfa46af9396f1a85025a331 to start an election
I20260807 03:39:58.420518 22984 raft_consensus.cc:3037] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:58.434779 18140 leader_election.cc:302] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
W20260807 03:39:58.487561 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 52902.3us
I20260807 03:39:58.488910 18140 leader_election.cc:302] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:58.460031 18190 raft_consensus.cc:1226] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 7. Current term is 8. Ops: []
I20260807 03:39:58.436259 23041 raft_consensus.cc:2781] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260807 03:39:58.465564 18189 raft_consensus.cc:1226] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 8. Current term is 9. Ops: []
I20260807 03:39:58.465969 22862 raft_consensus.cc:686] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:58.468914 23150 raft_consensus.cc:3032] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Stepping down as leader of term 7
I20260807 03:39:58.470820 22762 raft_consensus.cc:3032] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 03:39:58.454336 18310 raft_consensus.cc:2454] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 9.
I20260807 03:39:58.506319 18141 leader_election.cc:302] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:58.455377 18188 raft_consensus.cc:2454] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 9.
I20260807 03:39:58.490499 18140 leader_election.cc:302] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:58.493552 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
W20260807 03:39:58.495056 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 241127us
I20260807 03:39:58.447576 23257 raft_consensus.cc:686] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [term 7 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:58.499015 18187 raft_consensus.cc:2454] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 5.
I20260807 03:39:58.414932 23158 consensus_queue.cc:237] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:58.515810 18249 leader_election.cc:302] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:58.516709 22881 raft_consensus.cc:2781] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260807 03:39:58.498096 23041 raft_consensus.cc:482] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:58.541826 23041 raft_consensus.cc:3037] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Advancing to term 10
I20260807 03:39:58.542843 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:58.524941 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:58.499769 23248 raft_consensus.cc:2726] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260807 03:39:58.531358 18298 raft_consensus.cc:1226] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 7. Current term is 8. Ops: []
I20260807 03:39:58.497888 22476 raft_consensus.cc:2726] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260807 03:39:58.556092 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:58.556667 22932 raft_consensus.cc:504] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:58.531723 18183 raft_consensus.cc:2454] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 7.
I20260807 03:39:58.559017 22932 leader_election.cc:288] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 11 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:58.563143 23306 log.cc:924] Time spent T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.132s user 0.003s sys 0.000s
I20260807 03:39:58.551373 22917 raft_consensus.cc:504] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:58.567931 22917 leader_election.cc:288] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:58.517768 18140 leader_election.cc:302] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:58.532572 22896 raft_consensus.cc:2781] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader election won for term 9
I20260807 03:39:58.569389 22896 raft_consensus.cc:686] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:58.505292 22762 raft_consensus.cc:729] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:58.570925 22762 consensus_queue.cc:260] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:58.572445 22476 raft_consensus.cc:2781] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader election won for term 6
W20260807 03:39:58.572535 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 54870.7us
I20260807 03:39:58.573093 22476 raft_consensus.cc:686] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:58.573527 22427 raft_consensus.cc:3032] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 9 LEADER]: Stepping down as leader of term 9
I20260807 03:39:58.533188 23243 raft_consensus.cc:2726] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260807 03:39:58.573921 22427 raft_consensus.cc:729] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:58.573791 23248 consensus_queue.cc:1046] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 03:39:58.522316 22881 raft_consensus.cc:482] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:58.574955 22762 raft_consensus.cc:3037] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:58.534301 19287 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf"
dest_uuid: "5c9b5062bdfa46af9396f1a85025a331"
from {username='slave'} at 127.0.0.1:59906
I20260807 03:39:58.577450 19287 raft_consensus.cc:482] T 6b66966247c147308a4b8488557e0bdf P 5c9b5062bdfa46af9396f1a85025a331 [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:39:58.577821 19287 raft_consensus.cc:3037] T 6b66966247c147308a4b8488557e0bdf P 5c9b5062bdfa46af9396f1a85025a331 [term 9 FOLLOWER]: Advancing to term 10
I20260807 03:39:58.574846 23145 raft_consensus.cc:3032] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 LEADER]: Stepping down as leader of term 9
I20260807 03:39:58.580987 23145 raft_consensus.cc:729] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:58.540241 18249 leader_election.cc:302] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:58.540774 19297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331"
I20260807 03:39:58.540906 23261 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:58.500514 23150 raft_consensus.cc:729] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:58.448555 23302 raft_consensus.cc:504] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:58.549837 18295 raft_consensus.cc:1226] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:58.555058 23270 leader_election.cc:288] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:58.533070 22862 consensus_queue.cc:237] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:58.575073 22881 raft_consensus.cc:3037] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Advancing to term 10
I20260807 03:39:58.539088 23001 leader_election.cc:288] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 election: Requested vote from peers 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:58.535983 23257 consensus_queue.cc:237] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:58.583580 18140 leader_election.cc:302] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
W20260807 03:39:58.620075 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 43743.3us
I20260807 03:39:58.621001 18140 leader_election.cc:302] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:58.622267 18140 leader_election.cc:302] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
W20260807 03:39:58.584658 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 444568us
I20260807 03:39:58.624251 18248 leader_election.cc:302] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:58.625582 18248 leader_election.cc:302] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:58.570262 22897 raft_consensus.cc:3032] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 LEADER]: Stepping down as leader of term 5
W20260807 03:39:58.591431 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 51261.5us
I20260807 03:39:58.609087 22847 raft_consensus.cc:2781] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 03:39:58.579506 23137 raft_consensus.cc:504] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:58.562732 22984 raft_consensus.cc:504] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:58.605948 23314 log.cc:924] Time spent T a3dc635ee56b4559a87f5955bb83235a P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.142s user 0.002s sys 0.005s
I20260807 03:39:58.627751 18249 leader_election.cc:302] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:58.592633 18138 leader_election.cc:302] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49
W20260807 03:39:58.637795 18138 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.132:41381, user_credentials={real_user=slave}} blocked reactor thread for 45259.8us
I20260807 03:39:58.629748 23316 raft_consensus.cc:2726] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:58.638553 23201 raft_consensus.cc:2726] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260807 03:39:58.642143 23137 leader_election.cc:288] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 11 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:58.628079 23001 raft_consensus.cc:2781] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader election won for term 9
I20260807 03:39:58.642979 19287 raft_consensus.cc:504] T 6b66966247c147308a4b8488557e0bdf P 5c9b5062bdfa46af9396f1a85025a331 [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:58.630092 22897 raft_consensus.cc:729] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:58.645370 22916 consensus_queue.cc:1046] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:39:58.574836 22427 consensus_queue.cc:260] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:58.636034 22530 raft_consensus.cc:2726] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:58.648041 23302 leader_election.cc:288] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:58.608631 23248 consensus_queue.cc:1046] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.024s
I20260807 03:39:58.654541 23248 raft_consensus.cc:3032] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 LEADER]: Stepping down as leader of term 7
I20260807 03:39:58.654906 23248 raft_consensus.cc:729] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:58.631623 18140 leader_election.cc:302] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:58.655630 23229 raft_consensus.cc:2726] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260807 03:39:58.657176 18140 leader_election.cc:398] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 election: Vote denied by peer d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163) with higher term. Message: Invalid argument: T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for earlier term 7. Current term is 8.
W20260807 03:39:58.658036 23305 log.cc:924] Time spent T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.197s user 0.001s sys 0.002s
I20260807 03:39:58.658245 23229 consensus_queue.cc:1046] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260807 03:39:58.659447 23254 raft_consensus.cc:3032] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 10 LEADER]: Stepping down as leader of term 10
I20260807 03:39:58.635316 18248 leader_election.cc:302] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:58.627241 23270 consensus_queue.cc:1046] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
I20260807 03:39:58.659926 23254 raft_consensus.cc:729] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:58.660986 23254 consensus_queue.cc:260] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:58.662240 22897 consensus_queue.cc:260] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:58.662812 23254 raft_consensus.cc:3037] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Advancing to term 11
I20260807 03:39:58.663722 22897 raft_consensus.cc:3037] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:58.645242 19297 raft_consensus.cc:1261] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 5 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260807 03:39:58.665932 19297 pending_rounds.cc:76] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331: Aborting all ops after (but not including) 5
I20260807 03:39:58.666178 19297 pending_rounds.cc:97] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260807 03:39:58.666424 19297 raft_consensus.cc:2866] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 03:39:58.636772 23045 raft_consensus.cc:2726] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 03:39:58.668265 22847 raft_consensus.cc:686] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:58.669214 22847 consensus_queue.cc:237] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:58.670625 22984 leader_election.cc:288] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:58.674206 22427 raft_consensus.cc:3037] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Advancing to term 10
I20260807 03:39:58.675069 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:58.675172 23247 consensus_queue.cc:1035] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 03:39:58.643325 23001 raft_consensus.cc:686] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:58.676699 23150 consensus_queue.cc:260] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:58.677914 23001 consensus_queue.cc:237] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:58.680373 23116 raft_consensus.cc:504] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:58.681538 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 53860.7us
I20260807 03:39:58.682349 23116 leader_election.cc:288] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:58.574900 22441 raft_consensus.cc:504] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:58.683655 18140 leader_election.cc:302] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:58.684386 22441 leader_election.cc:288] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:58.570283 22896 consensus_queue.cc:237] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:58.643396 23041 raft_consensus.cc:504] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:58.685104 23243 raft_consensus.cc:2726] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260807 03:39:58.686790 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 51570.8us
I20260807 03:39:58.687395 23261 raft_consensus.cc:2781] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 03:39:58.687471 18248 leader_election.cc:302] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:58.687920 23261 raft_consensus.cc:482] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:58.688261 23045 raft_consensus.cc:2726] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 03:39:58.573932 22476 consensus_queue.cc:237] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:58.689769 23243 raft_consensus.cc:2781] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Leader election won for term 10
I20260807 03:39:58.690464 23243 raft_consensus.cc:686] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [term 10 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:58.682482 18249 leader_election.cc:302] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:58.691675 18193 raft_consensus.cc:1226] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 8. Current term is 9. Ops: []
I20260807 03:39:58.688328 23261 raft_consensus.cc:3037] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Advancing to term 9
W20260807 03:39:58.627193 19240 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 229991us
I20260807 03:39:58.695643 18297 raft_consensus.cc:1261] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260807 03:39:58.690728 17912 catalog_manager.cc:5495] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331 reported cstate change: term changed from 3 to 5, leader changed from d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) to 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132). New cstate: current_term: 5 leader_uuid: "5c9b5062bdfa46af9396f1a85025a331" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 03:39:58.701016 23045 raft_consensus.cc:2781] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader election won for term 9
I20260807 03:39:58.700429 18140 leader_election.cc:302] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:58.705797 23201 raft_consensus.cc:2726] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 03:39:58.707154 18140 leader_election.cc:302] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:58.699393 18248 leader_election.cc:302] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:58.708024 23201 raft_consensus.cc:2726] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 03:39:58.709117 18140 leader_election.cc:302] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:58.711483 18140 leader_election.cc:302] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:58.677945 23150 raft_consensus.cc:3037] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Advancing to term 8
W20260807 03:39:58.697276 23133 consensus_meta.cc:287] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331: Time spent flushing consensus metadata: real 0.654s user 0.013s sys 0.000s
W20260807 03:39:58.714105 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 308752us
I20260807 03:39:58.713582 18140 leader_election.cc:302] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:58.684434 23229 raft_consensus.cc:2726] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260807 03:39:58.721182 17912 catalog_manager.cc:5495] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "66a36e1ac8e945e893668e282d394c49" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:58.726971 17915 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39494 (request call id 198) took 5398 ms. Trace:
I20260807 03:39:58.719395 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:58.733875 18140 leader_election.cc:302] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:58.738867 17912 catalog_manager.cc:5495] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "5c9b5062bdfa46af9396f1a85025a331" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 03:39:58.655718 23248 consensus_queue.cc:260] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:58.743155 17912 catalog_manager.cc:5495] T 4d3a25995c8341c0bf39232774e3a917 P 5c9b5062bdfa46af9396f1a85025a331 reported cstate change: term changed from 6 to 8, leader changed from d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129) to 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132). New cstate: current_term: 8 leader_uuid: "5c9b5062bdfa46af9396f1a85025a331" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 03:39:58.744020 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:58.722956 23270 raft_consensus.cc:2781] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:39:58.748008 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:58.713272 22441 consensus_queue.cc:1035] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.029s
I20260807 03:39:58.730574 23045 raft_consensus.cc:686] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [term 9 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:58.719560 23201 raft_consensus.cc:2781] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader election won for term 9
I20260807 03:39:58.751452 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:58.760254 18180 raft_consensus.cc:2362] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 5. Re-sending same reply.
I20260807 03:39:58.749002 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:58.753525 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922e1d3cde8f476ea7780363c6b8693c" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:58.760864 18249 leader_election.cc:302] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:58.754678 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:58.761992 18189 raft_consensus.cc:2354] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 9. Current term is 10.
W20260807 03:39:58.753394 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 54003.5us
I20260807 03:39:58.764931 18248 leader_election.cc:302] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:39:58.637027 23145 consensus_queue.cc:260] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:58.781723 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:39:58.756520 17899 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.LeaderStepDown -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 270318us
I20260807 03:39:58.740728 23271 raft_consensus.cc:2781] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:39:58.736645 23116 consensus_queue.cc:1046] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.024s
I20260807 03:39:58.813058 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: true dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:58.815531 16154 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 03:39:58.827858 18075 raft_consensus.cc:3037] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Advancing to term 10
I20260807 03:39:58.755829 18027 leader_election.cc:302] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:58.780938 18249 leader_election.cc:302] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:58.750554 23270 raft_consensus.cc:482] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:58.749897 22530 raft_consensus.cc:2726] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260807 03:39:58.731662 17915 rpcz_store.cc:276] 0807 03:39:53.328467 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:53.328717 (+ 250us) service_pool.cc:227] Handling call
0807 03:39:58.726881 (+5398164us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 03:39:57.108132 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 03:39:57.108683 (+ 551us) write_op.cc:432] Acquiring schema lock in shared mode
0807 03:39:57.108735 (+ 52us) write_op.cc:435] Acquired schema lock
0807 03:39:57.108750 (+ 15us) tablet.cc:660] Decoding operations
0807 03:39:57.109727 (+ 977us) write_op.cc:620] Acquiring the partition lock for write op
0807 03:39:57.109786 (+ 59us) write_op.cc:641] Partition lock acquired for write op
0807 03:39:57.109814 (+ 28us) tablet.cc:683] Acquiring locks for 10 operations
0807 03:39:57.221933 (+112119us) tablet.cc:699] Row locks acquired
0807 03:39:57.241191 (+ 19258us) write_op.cc:260] PREPARE: finished
0807 03:39:57.276238 (+ 35047us) write_op.cc:270] Start()
0807 03:39:57.287726 (+ 11488us) write_op.cc:276] Timestamp: P: 1786073997273554 usec, L: 0
0807 03:39:57.288645 (+ 919us) op_driver.cc:331] REPLICATION: starting
0807 03:39:57.404356 (+115711us) log.cc:842] Serialized 4119 byte log entry
0807 03:39:57.849967 (+445611us) op_driver.cc:444] REPLICATION: finished
0807 03:39:58.189400 (+339433us) write_op.cc:301] APPLY: starting
0807 03:39:58.189492 (+ 92us) tablet.cc:1365] starting BulkCheckPresence
0807 03:39:58.189736 (+ 244us) tablet.cc:1368] finished BulkCheckPresence
0807 03:39:58.189873 (+ 137us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 03:39:58.190979 (+ 1106us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 03:39:58.191268 (+ 289us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=10
0807 03:39:58.191319 (+ 51us) write_op.cc:312] APPLY: finished
0807 03:39:58.191648 (+ 329us) log.cc:842] Serialized 77 byte log entry
0807 03:39:58.191823 (+ 175us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:39:58.191950 (+ 127us) write_op.cc:454] Released schema lock
0807 03:39:58.191999 (+ 49us) write_op.cc:341] FINISH: Updating metrics
0807 03:39:58.240058 (+ 48059us) write_op.cc:489] Releasing partition, row and schema locks
0807 03:39:58.243716 (+ 3658us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":20384,"reported_tablets":20,"child_traces":[["op",{"apply.queue_time_us":296254,"apply.run_cpu_time_us":3748,"apply.run_wall_time_us":103548,"num_ops":10,"prepare.queue_time_us":348006,"prepare.run_cpu_time_us":12315,"prepare.run_wall_time_us":462859,"raft.queue_time_us":235353,"raft.run_cpu_time_us":47864,"raft.run_wall_time_us":158089,"replication_time_us":561233,"thread_start_us":718529,"threads_started":4,"wal-append.queue_time_us":207998}]]}
I20260807 03:39:58.839190 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba8413916634e039111c838fbe1e587" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:58.769758 23132 raft_consensus.cc:3037] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Advancing to term 11
I20260807 03:39:58.863960 18072 raft_consensus.cc:2454] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
W20260807 03:39:58.862367 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 106640us
I20260807 03:39:58.844504 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:58.850343 18081 raft_consensus.cc:1226] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 7. Current term is 8. Ops: []
I20260807 03:39:58.847651 18082 raft_consensus.cc:1226] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20260807 03:39:58.760396 23316 raft_consensus.cc:2781] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 03:39:58.851972 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:58.882558 18071 raft_consensus.cc:2454] T 67de6ea5223f4eefa2edd875f9dfd1bc P d9716e2bf06442f5b3b0e81087c53c4d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 13.
I20260807 03:39:58.853732 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:58.855547 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:58.884356 18070 raft_consensus.cc:2379] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 9: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:58.872432 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:58.873171 18298 raft_consensus.cc:1226] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 8. Current term is 9. Ops: []
I20260807 03:39:58.873867 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:58.874691 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:58.753118 22984 consensus_queue.cc:1046] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.077s
I20260807 03:39:58.880770 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:58.887614 18309 raft_consensus.cc:3037] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Advancing to term 10
I20260807 03:39:58.890439 18073 raft_consensus.cc:1226] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 6. Current term is 7. Ops: []
W20260807 03:39:58.850499 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 69659.4us
I20260807 03:39:58.898116 18075 raft_consensus.cc:2454] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 10.
I20260807 03:39:58.772403 23248 raft_consensus.cc:3037] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:58.827461 23271 raft_consensus.cc:482] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:58.899369 23045 consensus_queue.cc:237] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:58.899966 23269 raft_consensus.cc:2781] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader election won for term 8
I20260807 03:39:58.772047 23201 raft_consensus.cc:686] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:58.759394 22932 consensus_queue.cc:1046] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.129s
I20260807 03:39:58.902024 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:58.905408 18309 raft_consensus.cc:2454] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 10.
I20260807 03:39:58.932637 18297 raft_consensus.cc:3037] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Advancing to term 11
I20260807 03:39:58.858171 23318 raft_consensus.cc:2781] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 03:39:58.771013 19289 raft_consensus.cc:1226] T 6b66966247c147308a4b8488557e0bdf P 5c9b5062bdfa46af9396f1a85025a331 [term 10 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 9. Current term is 10. Ops: []
I20260807 03:39:58.908077 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:58.916963 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ca0223f47b46fd955704eba5259370" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:58.771377 23041 leader_election.cc:288] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 10 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:58.863194 23270 raft_consensus.cc:3037] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:58.929601 18293 raft_consensus.cc:1261] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260807 03:39:58.879551 22426 raft_consensus.cc:2781] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader election won for term 9
I20260807 03:39:58.819757 22881 raft_consensus.cc:504] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:58.938571 18188 raft_consensus.cc:2379] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 10: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:58.943111 18190 raft_consensus.cc:1226] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [term 8 LEADER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 7. Current term is 8. Ops: []
I20260807 03:39:58.924255 23271 raft_consensus.cc:3037] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:58.768221 23243 consensus_queue.cc:237] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:58.921567 23316 raft_consensus.cc:686] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:58.834159 23284 consensus_queue.cc:1046] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.128s
I20260807 03:39:58.942538 23318 raft_consensus.cc:686] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [term 7 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:58.934301 23145 raft_consensus.cc:3037] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Advancing to term 10
I20260807 03:39:58.920356 22984 consensus_queue.cc:1046] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20260807 03:39:58.990633 22984 raft_consensus.cc:3032] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 8 LEADER]: Stepping down as leader of term 8
I20260807 03:39:58.990978 22984 raft_consensus.cc:729] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:58.937383 23269 raft_consensus.cc:686] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:58.966542 23316 consensus_queue.cc:237] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:58.944295 22476 consensus_queue.cc:1035] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
W20260807 03:39:59.011814 23329 log.cc:924] Time spent T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20260807 03:39:59.036378 23269 consensus_queue.cc:237] T fe8bab68c1264fc8a2c4164e0f17f9b6 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:59.040369 23318 consensus_queue.cc:237] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:59.031576 23261 raft_consensus.cc:504] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:59.046613 23261 leader_election.cc:288] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:59.053261 23271 raft_consensus.cc:504] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:59.129679 23271 leader_election.cc:288] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:59.129700 22881 leader_election.cc:288] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:59.096342 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:59.099256 18191 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44660 (request call id 2601) took 1020 ms. Trace:
I20260807 03:39:59.132771 22984 consensus_queue.cc:260] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:59.135511 22984 raft_consensus.cc:3037] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:59.104992 22862 raft_consensus.cc:482] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:59.106060 23207 raft_consensus.cc:3032] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 03:39:59.108572 23013 raft_consensus.cc:3032] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 LEADER]: Stepping down as leader of term 8
I20260807 03:39:59.117863 18183 raft_consensus.cc:1226] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 LEADER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 8. Current term is 9. Ops: []
I20260807 03:39:59.124600 18193 raft_consensus.cc:2396] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260807 03:39:58.976008 23270 raft_consensus.cc:504] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:58.949689 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 184833us
I20260807 03:39:59.133108 18191 rpcz_store.cc:276] 0807 03:39:58.078395 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:58.488160 (+409765us) service_pool.cc:227] Handling call
0807 03:39:58.510712 (+ 22552us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:58.511213 (+ 501us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:58.511253 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:39:58.528279 (+ 17026us) log.cc:842] Serialized 572 byte log entry
0807 03:39:58.810483 (+282204us) raft_consensus.cc:1631] Marking committed up to 8
0807 03:39:58.810532 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:58.810632 (+ 100us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:39:59.098446 (+287814us) raft_consensus.cc:1671] finished
0807 03:39:59.098498 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:39:59.098935 (+ 437us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":281854,"threads_started":1,"wal-append.queue_time_us":395973}
I20260807 03:39:59.161602 23013 raft_consensus.cc:729] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:58.952960 18293 raft_consensus.cc:1226] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 3. Current term is 4. Ops: []
I20260807 03:39:59.103472 18295 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf"
dest_uuid: "66a36e1ac8e945e893668e282d394c49"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57260
I20260807 03:39:59.159634 23207 raft_consensus.cc:729] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:59.219614 22862 raft_consensus.cc:504] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:39:59.221207 23013 consensus_queue.cc:260] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:58.994206 23201 consensus_queue.cc:237] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:59.223826 22862 leader_election.cc:288] T d416e1015a2644958b0c8f9147d403a2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:59.258220 18295 raft_consensus.cc:595] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 9 LEADER]: Received request to transfer leadership
W20260807 03:39:59.274570 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 403671us
I20260807 03:39:59.274819 22426 raft_consensus.cc:686] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [term 9 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:59.283346 23207 consensus_queue.cc:260] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:59.284969 23207 raft_consensus.cc:3037] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
W20260807 03:39:59.294219 18027 leader_election.cc:334] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057): Timed out: RequestConsensusVote RPC to 127.15.198.130:37057 timed out after 1.565s (SENT)
W20260807 03:39:59.291491 22427 consensus_meta.cc:287] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.617s user 0.008s sys 0.008s
I20260807 03:39:59.294827 18027 leader_election.cc:302] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:59.300951 19287 leader_election.cc:288] T 6b66966247c147308a4b8488557e0bdf P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 10 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:59.306356 23021 raft_consensus.cc:3032] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 8 LEADER]: Stepping down as leader of term 8
I20260807 03:39:59.307051 23021 raft_consensus.cc:729] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
W20260807 03:39:59.306597 18003 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 6b66966247c147308a4b8488557e0bdf already in progress
W20260807 03:39:59.315272 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 68657.9us
W20260807 03:39:59.309248 18003 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 6b66966247c147308a4b8488557e0bdf already in progress
I20260807 03:39:59.309195 23013 raft_consensus.cc:3037] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:59.358458 23021 consensus_queue.cc:260] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:59.361123 23021 raft_consensus.cc:3037] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:59.363001 22426 consensus_queue.cc:237] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:39:59.370497 23150 consensus_meta.cc:287] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.646s user 0.010s sys 0.005s
I20260807 03:39:59.373498 23351 raft_consensus.cc:482] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 5c9b5062bdfa46af9396f1a85025a331)
I20260807 03:39:59.374274 23351 raft_consensus.cc:504] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:59.374861 23270 leader_election.cc:288] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:59.376714 23351 leader_election.cc:288] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:59.379722 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:59.379927 18189 raft_consensus.cc:1226] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260807 03:39:59.381310 18188 raft_consensus.cc:1226] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 LEADER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 8. Current term is 9. Ops: []
I20260807 03:39:59.381525 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b69f685f5d40edbdd2981d75bc2ad2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:59.382299 18191 raft_consensus.cc:3037] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Advancing to term 11
W20260807 03:39:59.383049 19239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 44625.6us
I20260807 03:39:59.384611 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:59.385643 18183 raft_consensus.cc:2379] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 9: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:59.392134 18184 raft_consensus.cc:2454] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 10.
I20260807 03:39:59.392688 23167 consensus_queue.cc:1046] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260807 03:39:59.394263 22897 consensus_meta.cc:287] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 0.632s user 0.008s sys 0.005s
I20260807 03:39:59.394019 23167 consensus_queue.cc:1035] T a3dc635ee56b4559a87f5955bb83235a P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 03:39:59.394647 23302 raft_consensus.cc:3032] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:39:59.395098 23302 raft_consensus.cc:729] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5c9b5062bdfa46af9396f1a85025a331, State: Running, Role: LEADER
W20260807 03:39:59.396909 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 102771us
I20260807 03:39:59.398151 23010 raft_consensus.cc:3032] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 8 LEADER]: Stepping down as leader of term 8
I20260807 03:39:59.398927 23010 raft_consensus.cc:729] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:59.398265 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de9873c6144d36b33a1a666691869a" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:59.399596 18027 leader_election.cc:302] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:59.399840 18183 raft_consensus.cc:3037] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:59.400656 23269 raft_consensus.cc:482] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:59.411043 18027 leader_election.cc:302] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:39:59.412312 23269 raft_consensus.cc:504] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:59.412912 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:59.388890 23318 raft_consensus.cc:2932] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 9 LEADER]: Committing config change with OpId 9.9: config changed from index 7 to 9, 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } } }
I20260807 03:39:59.429752 23319 raft_consensus.cc:2726] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 03:39:59.430529 23041 raft_consensus.cc:2726] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260807 03:39:59.433285 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:59.434917 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:59.435099 18140 leader_election.cc:302] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:59.435601 18298 raft_consensus.cc:1261] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260807 03:39:59.401130 22862 raft_consensus.cc:2726] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260807 03:39:59.436475 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 47587.2us
I20260807 03:39:59.434960 18311 raft_consensus.cc:1226] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 9. Current term is 10. Ops: []
I20260807 03:39:59.427608 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:59.457532 18310 raft_consensus.cc:1261] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260807 03:39:59.458715 18297 raft_consensus.cc:2454] T 23b69f685f5d40edbdd2981d75bc2ad2 P 66a36e1ac8e945e893668e282d394c49 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 11.
I20260807 03:39:59.462690 18298 raft_consensus.cc:1226] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 7. Current term is 8. Ops: []
I20260807 03:39:59.451510 22476 raft_consensus.cc:2741] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260807 03:39:59.465539 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2371c1d6174640aa9a9ee19732634a88" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:59.466045 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a34af1cebc14f92bad043893aa1f964" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:59.466346 18082 raft_consensus.cc:3037] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Advancing to term 10
I20260807 03:39:59.466753 18073 raft_consensus.cc:3037] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:59.467038 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:59.466353 18071 raft_consensus.cc:1261] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260807 03:39:59.468083 18311 raft_consensus.cc:1261] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260807 03:39:59.469573 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:59.475591 23269 leader_election.cc:288] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:59.451588 23316 raft_consensus.cc:482] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:59.476125 18069 raft_consensus.cc:1226] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260807 03:39:59.476490 23243 raft_consensus.cc:482] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:59.476704 23316 raft_consensus.cc:504] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:59.477314 23243 raft_consensus.cc:504] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:59.477705 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:59.478477 18081 raft_consensus.cc:3037] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:59.479187 23316 leader_election.cc:288] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:59.479530 23243 leader_election.cc:288] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:59.480556 23153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d416e1015a2644958b0c8f9147d403a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:39:59.480660 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33252bff379e4392a9a815ecdbc447a7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:59.481665 18080 raft_consensus.cc:2379] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 9: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:59.482686 18072 raft_consensus.cc:2379] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 10: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:59.488925 23321 raft_consensus.cc:482] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:59.489595 23321 raft_consensus.cc:504] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:59.491264 23354 raft_consensus.cc:482] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:39:59.491696 23264 raft_consensus.cc:482] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 5c9b5062bdfa46af9396f1a85025a331)
W20260807 03:39:59.492630 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 50942.8us
I20260807 03:39:59.494853 18293 raft_consensus.cc:1261] T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260807 03:39:59.426009 23010 consensus_queue.cc:260] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:59.508376 18191 raft_consensus.cc:2454] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 11.
I20260807 03:39:59.514906 23201 raft_consensus.cc:482] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:59.515239 23264 raft_consensus.cc:504] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:59.515594 23201 raft_consensus.cc:504] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:59.515305 18305 raft_consensus.cc:1261] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260807 03:39:59.516608 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:59.518078 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: true dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:59.525005 18075 raft_consensus.cc:1261] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260807 03:39:59.525902 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:59.419663 23302 consensus_queue.cc:260] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:59.491989 23321 leader_election.cc:288] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:59.534296 18183 raft_consensus.cc:2454] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 6.
W20260807 03:39:59.536293 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 89056.2us
I20260807 03:39:59.538411 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: true dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:59.538558 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:59.539278 18082 raft_consensus.cc:2454] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 10.
I20260807 03:39:59.543426 23302 raft_consensus.cc:3037] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:59.543893 23010 raft_consensus.cc:3037] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:59.547716 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ca0223f47b46fd955704eba5259370" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:59.548760 18080 raft_consensus.cc:1226] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 11 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 10. Current term is 11. Ops: []
I20260807 03:39:59.548998 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:59.548887 18069 raft_consensus.cc:1226] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:59.555716 23133 raft_consensus.cc:482] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:59.570892 23133 raft_consensus.cc:504] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:59.572219 18027 leader_election.cc:302] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:59.572927 18184 raft_consensus.cc:1226] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 8. Current term is 9. Ops: []
I20260807 03:39:59.573885 18027 leader_election.cc:302] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:59.574810 23269 raft_consensus.cc:2781] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:39:59.575201 23269 raft_consensus.cc:482] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:59.575614 23269 raft_consensus.cc:3037] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:39:59.577059 18069 raft_consensus.cc:1261] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 03:39:59.578481 18069 raft_consensus.cc:1261] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260807 03:39:59.581336 18027 leader_election.cc:302] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:59.585547 18187 raft_consensus.cc:1226] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:59.558964 23201 leader_election.cc:288] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:59.541587 22426 consensus_queue.cc:1046] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.018s
I20260807 03:39:59.559859 18071 raft_consensus.cc:1226] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:59.562964 18190 raft_consensus.cc:1226] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 9. Current term is 10. Ops: []
I20260807 03:39:59.568920 18193 raft_consensus.cc:1226] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260807 03:39:59.585604 18140 leader_election.cc:302] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:59.599830 18071 raft_consensus.cc:1261] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 03:39:59.552779 18300 raft_consensus.cc:2454] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 5.
I20260807 03:39:59.589862 18072 raft_consensus.cc:2379] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 6: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:59.610462 18071 raft_consensus.cc:1226] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 9. Current term is 10. Ops: []
I20260807 03:39:59.611613 18071 raft_consensus.cc:1226] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 9. Current term is 10. Ops: []
I20260807 03:39:59.612263 22862 raft_consensus.cc:2726] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 03:39:59.491900 23354 raft_consensus.cc:504] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:59.591212 23041 raft_consensus.cc:2726] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260807 03:39:59.614493 23354 leader_election.cc:288] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:59.592033 22476 consensus_queue.cc:1046] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 03:39:59.593261 23376 raft_consensus.cc:482] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:59.615765 22476 raft_consensus.cc:3032] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 LEADER]: Stepping down as leader of term 9
I20260807 03:39:59.615391 23376 raft_consensus.cc:504] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:59.616217 22476 raft_consensus.cc:729] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:59.617326 22476 consensus_queue.cc:260] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:59.617885 23376 leader_election.cc:288] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:59.618652 22476 raft_consensus.cc:3037] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Advancing to term 10
I20260807 03:39:59.597417 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:59.620820 18080 raft_consensus.cc:2379] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 10: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:59.600435 18191 raft_consensus.cc:3037] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Advancing to term 10
I20260807 03:39:59.600625 23201 raft_consensus.cc:2781] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260807 03:39:59.621628 23201 raft_consensus.cc:482] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:59.622032 23201 raft_consensus.cc:3037] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 13 FOLLOWER]: Advancing to term 14
I20260807 03:39:59.600463 18140 leader_election.cc:302] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:59.625188 18140 leader_election.cc:302] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:59.626012 22897 raft_consensus.cc:2726] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260807 03:39:59.626612 22897 raft_consensus.cc:2726] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 03:39:59.627247 22897 consensus_queue.cc:1046] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 03:39:59.628397 22897 raft_consensus.cc:3032] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Stepping down as leader of term 6
I20260807 03:39:59.628754 22897 raft_consensus.cc:729] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:59.629734 22897 consensus_queue.cc:260] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:59.630957 22897 raft_consensus.cc:3037] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:59.631837 18140 leader_election.cc:302] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:59.633931 18075 raft_consensus.cc:1226] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260807 03:39:59.634788 18082 raft_consensus.cc:1226] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 9. Current term is 10. Ops: []
I20260807 03:39:59.636716 23264 leader_election.cc:288] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
W20260807 03:39:59.604547 23132 consensus_meta.cc:287] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.715s user 0.008s sys 0.006s
I20260807 03:39:59.606590 18079 raft_consensus.cc:1261] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260807 03:39:59.637508 23132 raft_consensus.cc:2726] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260807 03:39:59.612236 18246 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.132:41381, user_credentials={real_user=slave}} blocked reactor thread for 58774.3us
I20260807 03:39:59.638398 22441 raft_consensus.cc:2781] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Leader election won for term 10
I20260807 03:39:59.638957 22441 raft_consensus.cc:686] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 10 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:59.640074 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:59.639941 22441 consensus_queue.cc:237] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:59.640731 18190 raft_consensus.cc:2454] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 10.
I20260807 03:39:59.641948 18183 raft_consensus.cc:1261] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260807 03:39:59.642840 18188 raft_consensus.cc:1226] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 6. Current term is 7. Ops: []
I20260807 03:39:59.644059 18187 raft_consensus.cc:1226] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:59.644140 19289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:59.588702 18081 raft_consensus.cc:2454] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 6.
I20260807 03:39:59.632529 23145 consensus_queue.cc:1035] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 03:39:59.590646 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a59a83a64df475ea15dd572f84024bd" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:59.641993 18193 raft_consensus.cc:1261] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 03:39:59.651103 23001 raft_consensus.cc:482] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:59.651641 23001 raft_consensus.cc:504] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:59.653355 23270 consensus_queue.cc:1046] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:39:59.654467 23270 consensus_queue.cc:1035] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 03:39:59.656059 23248 raft_consensus.cc:3032] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 03:39:59.656500 23248 raft_consensus.cc:729] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
W20260807 03:39:59.657460 22762 consensus_meta.cc:287] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 1.082s user 0.019s sys 0.004s
I20260807 03:39:59.657565 23248 consensus_queue.cc:260] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:59.659528 18310 raft_consensus.cc:1261] T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260807 03:39:59.663901 18073 raft_consensus.cc:2454] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 9.
I20260807 03:39:59.601974 23133 leader_election.cc:288] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:59.667055 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:59.668783 18081 raft_consensus.cc:1226] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 5. Current term is 6. Ops: []
I20260807 03:39:59.668473 18071 raft_consensus.cc:2454] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 11.
I20260807 03:39:59.670499 19239 leader_election.cc:302] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:59.671340 18079 raft_consensus.cc:1226] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 8. Current term is 9. Ops: []
I20260807 03:39:59.679067 18310 pending_rounds.cc:76] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49: Aborting all ops after (but not including) 8
I20260807 03:39:59.679366 18310 pending_rounds.cc:97] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49: Aborting uncommitted NO_OP operation due to leader change: 7.9
I20260807 03:39:59.679664 18310 raft_consensus.cc:2866] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 03:39:59.683547 19240 leader_election.cc:302] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:59.684621 23133 raft_consensus.cc:2781] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:39:59.685173 23133 raft_consensus.cc:482] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:39:59.685674 23133 raft_consensus.cc:3037] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:39:59.588084 18311 raft_consensus.cc:2454] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 5.
I20260807 03:39:59.715176 23376 raft_consensus.cc:2781] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260807 03:39:59.717618 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 85872.6us
I20260807 03:39:59.717885 23248 raft_consensus.cc:3037] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:39:59.717896 23376 raft_consensus.cc:482] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:59.718932 23376 raft_consensus.cc:3037] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [term 12 FOLLOWER]: Advancing to term 13
I20260807 03:39:59.719051 18189 raft_consensus.cc:1261] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260807 03:39:59.719873 18082 raft_consensus.cc:1261] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260807 03:39:59.718696 23270 consensus_queue.cc:1035] T 4cd2b7ebeed04ac6907b51b064e94894 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20260807 03:39:59.721895 23145 consensus_queue.cc:1035] T 7afbf00d2d944c18921cc32b7c48be16 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 03:39:59.722950 18140 leader_election.cc:302] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:59.725226 23261 raft_consensus.cc:2781] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 03:39:59.725795 23261 raft_consensus.cc:686] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:59.726872 23261 consensus_queue.cc:237] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:59.728313 22881 consensus_queue.cc:1035] T 79e3ecc1bf7d4ca9a7096616efc98585 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 03:39:59.728919 18140 leader_election.cc:302] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:59.729717 23270 raft_consensus.cc:2781] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader election won for term 9
W20260807 03:39:59.732800 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 151673us
W20260807 03:39:59.733405 19240 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 50162.5us
I20260807 03:39:59.730196 23270 raft_consensus.cc:686] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:59.730310 23376 raft_consensus.cc:504] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:59.736655 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:59.737526 18080 raft_consensus.cc:2379] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 7: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:59.737406 23270 consensus_queue.cc:237] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:59.727499 23201 raft_consensus.cc:504] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:59.738842 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:59.743098 18080 raft_consensus.cc:3037] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 11 FOLLOWER]: Advancing to term 13
I20260807 03:39:59.748919 18191 raft_consensus.cc:2454] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 10.
I20260807 03:39:59.762265 18027 leader_election.cc:302] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:59.764166 18298 raft_consensus.cc:3032] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 9 LEADER]: Stepping down as leader of term 9
I20260807 03:39:59.766114 19297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:59.767663 23013 raft_consensus.cc:2781] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 03:39:59.768123 18027 leader_election.cc:302] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:59.764549 18298 raft_consensus.cc:729] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:39:59.779264 18191 raft_consensus.cc:1226] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 9. Current term is 10. Ops: []
I20260807 03:39:59.785459 23041 raft_consensus.cc:2726] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:39:59.785538 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ea193a7604411b81684ccff437bc0a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:59.787102 18082 raft_consensus.cc:3037] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:39:59.769052 19289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3dc635ee56b4559a87f5955bb83235a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:59.789960 23134 consensus_queue.cc:1035] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.009s
I20260807 03:39:59.758512 23145 consensus_queue.cc:1035] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.021s
I20260807 03:39:59.791872 18075 raft_consensus.cc:1261] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260807 03:39:59.768129 23013 raft_consensus.cc:482] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:59.792240 23001 leader_election.cc:288] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:59.792415 23013 raft_consensus.cc:3037] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:39:59.794170 23207 raft_consensus.cc:482] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 5c9b5062bdfa46af9396f1a85025a331)
I20260807 03:39:59.795053 23001 raft_consensus.cc:482] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:59.794899 23207 raft_consensus.cc:504] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:59.779688 23318 raft_consensus.cc:482] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:59.796919 23318 raft_consensus.cc:504] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:59.799283 23318 leader_election.cc:288] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:59.800673 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:39:59.790318 22984 consensus_meta.cc:287] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.612s user 0.013s sys 0.000s
I20260807 03:39:59.747400 22881 consensus_queue.cc:1046] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20260807 03:39:59.764057 23376 leader_election.cc:288] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 13 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:39:59.795616 23001 raft_consensus.cc:504] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:59.827754 18082 raft_consensus.cc:2454] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 7.
I20260807 03:39:59.814049 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:59.831885 18075 raft_consensus.cc:1226] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 8. Current term is 9. Ops: []
I20260807 03:39:59.856276 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:59.860522 18079 raft_consensus.cc:1261] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260807 03:39:59.814424 23207 leader_election.cc:288] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:59.778657 22441 consensus_queue.cc:1035] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.075s
I20260807 03:39:59.866236 18072 raft_consensus.cc:3032] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 LEADER]: Stepping down as leader of term 8
I20260807 03:39:59.806399 22476 consensus_queue.cc:1035] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.049s
I20260807 03:39:59.866554 18072 raft_consensus.cc:729] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:39:59.871188 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f43f2cbf24aeca08087d9e2fb9b2a" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:59.873669 18298 consensus_queue.cc:260] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:59.874478 18191 raft_consensus.cc:2454] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 7.
I20260807 03:39:59.876405 18072 consensus_queue.cc:260] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:59.877377 18072 raft_consensus.cc:3037] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:39:59.877866 23001 leader_election.cc:288] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:39:59.880416 18248 leader_election.cc:302] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:39:59.881319 23270 consensus_queue.cc:1035] T 349190ae153f46c9a99c29df20734d3f P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:39:59.884665 23201 leader_election.cc:288] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 14 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:39:59.886000 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 118016us
I20260807 03:39:59.887655 18027 leader_election.cc:302] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:59.834966 23269 raft_consensus.cc:504] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:39:59.892045 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:59.892863 18187 raft_consensus.cc:2379] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 8: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:39:59.893592 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:59.894038 23269 leader_election.cc:288] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:39:59.877876 18140 leader_election.cc:302] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:39:59.900146 18069 raft_consensus.cc:1226] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 4. Current term is 5. Ops: []
I20260807 03:39:59.904837 18070 raft_consensus.cc:2454] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 9.
I20260807 03:39:59.864753 23376 raft_consensus.cc:482] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:39:59.909123 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:59.915593 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:39:59.916705 18310 raft_consensus.cc:2454] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 7.
I20260807 03:39:59.916735 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:39:59.917796 18193 raft_consensus.cc:1261] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 03:39:59.869256 22984 raft_consensus.cc:482] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:39:59.870651 19289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:59.922219 18081 raft_consensus.cc:1226] T 14dbf332d6394c66937326ef608dc8b5 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 LEADER]: Rejecting Update request from peer 3c4bceb836db4536a0f255fdf86abc02 for earlier term 7. Current term is 8. Ops: []
I20260807 03:39:59.918788 18073 raft_consensus.cc:1261] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260807 03:39:59.919550 18079 raft_consensus.cc:2454] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 6.
I20260807 03:39:59.925526 19287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:39:59.925933 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0047935053942d480774d46b081709d" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:39:59.926267 19287 raft_consensus.cc:2454] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 8.
I20260807 03:39:59.926664 18081 raft_consensus.cc:2379] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 11: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:39:59.927398 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:39:59.927911 18138 leader_election.cc:302] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331; no voters:
I20260807 03:39:59.928288 22881 raft_consensus.cc:2781] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 03:39:59.928563 23318 consensus_queue.cc:1046] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 03:39:59.929610 22426 consensus_queue.cc:1046] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 03:39:59.930105 23207 consensus_queue.cc:1046] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260807 03:39:59.917819 23010 raft_consensus.cc:2726] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260807 03:39:59.933959 18191 raft_consensus.cc:1226] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [term 10 LEADER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 9. Current term is 10. Ops: []
I20260807 03:39:59.933246 19240 leader_election.cc:302] T 6b66966247c147308a4b8488557e0bdf P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:39:59.918008 23134 raft_consensus.cc:3032] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 LEADER]: Stepping down as leader of term 7
I20260807 03:39:59.941880 18140 leader_election.cc:302] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
W20260807 03:39:59.942363 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 54826.6us
I20260807 03:39:59.918084 23354 raft_consensus.cc:2726] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260807 03:39:59.945426 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:59.948302 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:39:59.948067 18193 raft_consensus.cc:2454] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 8.
I20260807 03:39:59.957281 18072 raft_consensus.cc:1261] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 6. (term mismatch)
I20260807 03:39:59.937249 19289 raft_consensus.cc:2454] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 8.
I20260807 03:39:59.957731 18072 pending_rounds.cc:76] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d: Aborting all ops after (but not including) 5
I20260807 03:39:59.971832 18072 pending_rounds.cc:97] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d: Aborting uncommitted NO_OP operation due to leader change: 8.6
I20260807 03:39:59.972239 18072 raft_consensus.cc:2866] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 03:39:59.947160 23134 raft_consensus.cc:729] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:39:59.960367 23271 raft_consensus.cc:2781] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260807 03:39:59.973023 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 213849us
W20260807 03:39:59.974881 23375 log.cc:924] Time spent T 00000000000000000000000000000000 P 8da2065a77024f35bdfc0dc3adb83f69: Append to log took a long time: real 0.405s user 0.002s sys 0.000s
I20260807 03:39:59.976408 23346 consensus_queue.cc:1046] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
I20260807 03:39:59.918465 23133 raft_consensus.cc:504] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:59.973425 18249 leader_election.cc:302] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:39:59.971681 22896 consensus_queue.cc:1046] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.017s
I20260807 03:39:59.983682 19240 leader_election.cc:302] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331; no voters:
I20260807 03:39:59.984794 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6129789267024fac8a337070c10a472a" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:39:59.985347 18310 raft_consensus.cc:1261] T 33252bff379e4392a9a815ecdbc447a7 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260807 03:39:59.935933 23376 raft_consensus.cc:504] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:39:59.990173 18071 raft_consensus.cc:1261] T 0890f92b53f04149998f1f1c4e7ba60d P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260807 03:39:59.994791 18246 leader_election.cc:302] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49; no voters:
W20260807 03:39:59.996555 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 52293.2us
I20260807 03:39:59.998185 18249 leader_election.cc:302] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:40:00.000854 23167 raft_consensus.cc:2781] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 03:40:00.001112 18027 leader_election.cc:302] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:40:00.001768 23343 raft_consensus.cc:2781] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260807 03:40:00.002226 23343 raft_consensus.cc:482] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:40:00.002576 18027 leader_election.cc:302] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:40:00.002794 23207 consensus_queue.cc:1035] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:40:00.004269 23207 consensus_queue.cc:1046] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 03:40:00.005165 23207 consensus_queue.cc:1046] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 03:40:00.006145 23207 raft_consensus.cc:2726] T 9a59a83a64df475ea15dd572f84024bd P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 03:40:00.006749 23207 consensus_queue.cc:1035] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 03:40:00.006788 23013 raft_consensus.cc:504] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:00.007786 23150 raft_consensus.cc:3032] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 6 LEADER]: Stepping down as leader of term 6
I20260807 03:40:00.011651 23150 raft_consensus.cc:729] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:40:00.012586 23150 consensus_queue.cc:260] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:00.013978 23150 raft_consensus.cc:3037] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:40:00.008312 22426 raft_consensus.cc:3032] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 03:40:00.001300 23167 raft_consensus.cc:482] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:39:59.960227 23001 consensus_queue.cc:1035] T 1f41eb41bec8494c8a6708fbc44a22f1 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.051s
I20260807 03:40:00.009963 23351 consensus_queue.cc:1035] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 03:39:59.946421 22984 raft_consensus.cc:504] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
W20260807 03:40:00.015082 23372 log.cc:924] Time spent T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20260807 03:40:00.026773 23041 raft_consensus.cc:2726] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260807 03:40:00.026993 18248 leader_election.cc:302] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:40:00.032325 23010 raft_consensus.cc:2781] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader election won for term 6
I20260807 03:40:00.037410 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:40:00.038980 22984 leader_election.cc:288] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:40:00.039541 22984 raft_consensus.cc:2726] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260807 03:40:00.037444 23010 raft_consensus.cc:686] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:40:00.020241 18188 raft_consensus.cc:2454] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 12.
I20260807 03:40:00.003903 18027 leader_election.cc:302] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:40:00.011148 18249 leader_election.cc:302] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331
I20260807 03:40:00.045696 18249 leader_election.cc:302] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:40:00.047541 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
W20260807 03:40:00.018576 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 85640.9us
I20260807 03:40:00.035789 22441 raft_consensus.cc:482] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:40:00.061309 18080 raft_consensus.cc:2454] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 13.
I20260807 03:39:59.985059 23261 raft_consensus.cc:3037] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Advancing to term 11
I20260807 03:39:59.986016 23271 raft_consensus.cc:482] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 03:40:00.081990 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 95552us
W20260807 03:40:00.079273 18246 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.132:41381, user_credentials={real_user=slave}} blocked reactor thread for 84594.8us
I20260807 03:40:00.099480 18248 leader_election.cc:302] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:40:00.090368 18188 raft_consensus.cc:2379] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5c9b5062bdfa46af9396f1a85025a331 in current term 9: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:40:00.044618 23167 raft_consensus.cc:3037] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:40:00.002614 23343 raft_consensus.cc:3037] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Advancing to term 10
I20260807 03:40:00.015782 23346 consensus_queue.cc:1225] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Peer d9716e2bf06442f5b3b0e81087c53c4d log is divergent from this leader: its last log entry 8.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260807 03:40:00.028716 22980 raft_consensus.cc:3032] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 9 LEADER]: Stepping down as leader of term 9
I20260807 03:40:00.017099 22896 raft_consensus.cc:3032] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 8 LEADER]: Stepping down as leader of term 8
I20260807 03:40:00.025116 23376 leader_election.cc:288] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:40:00.042560 22426 raft_consensus.cc:729] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:40:00.019374 23133 leader_election.cc:288] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 6 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:40:00.142877 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 44546.6us
I20260807 03:40:00.107810 19239 leader_election.cc:302] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:40:00.077659 23122 raft_consensus.cc:2781] T 6b66966247c147308a4b8488557e0bdf P 5c9b5062bdfa46af9396f1a85025a331 [term 10 FOLLOWER]: Leader election won for term 10
I20260807 03:40:00.078191 23132 raft_consensus.cc:2726] T a21f43f2cbf24aeca08087d9e2fb9b2a P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260807 03:40:00.150521 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 70436.6us
I20260807 03:40:00.102840 23041 raft_consensus.cc:2781] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Leader election won for term 10
W20260807 03:40:00.153283 19239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 45559.5us
W20260807 03:40:00.153878 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 212059us
I20260807 03:40:00.158033 18140 leader_election.cc:302] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:40:00.159924 23122 raft_consensus.cc:686] T 6b66966247c147308a4b8488557e0bdf P 5c9b5062bdfa46af9396f1a85025a331 [term 10 LEADER]: Becoming Leader. State: Replica: 5c9b5062bdfa46af9396f1a85025a331, State: Running, Role: LEADER
I20260807 03:40:00.159539 18140 leader_election.cc:302] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:40:00.161394 23316 raft_consensus.cc:2726] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 03:40:00.161164 23122 consensus_queue.cc:237] T 6b66966247c147308a4b8488557e0bdf P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:40:00.162254 23316 consensus_queue.cc:1035] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 03:40:00.164536 22426 consensus_queue.cc:260] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:00.166114 22426 raft_consensus.cc:3037] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
W20260807 03:40:00.168678 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 69271.1us
I20260807 03:40:00.168803 23284 raft_consensus.cc:2781] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:40:00.169448 23284 raft_consensus.cc:482] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:40:00.169934 23284 raft_consensus.cc:3037] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:40:00.171236 18248 leader_election.cc:302] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:40:00.172395 23318 consensus_queue.cc:1046] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260807 03:40:00.098501 23021 raft_consensus.cc:2781] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 03:40:00.103797 23271 raft_consensus.cc:3037] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:40:00.140547 22980 raft_consensus.cc:729] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:40:00.145964 23346 raft_consensus.cc:482] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:40:00.144779 22881 raft_consensus.cc:686] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [term 7 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:40:00.149645 22896 raft_consensus.cc:729] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:40:00.201661 19297 raft_consensus.cc:1226] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 6 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 5. Current term is 6. Ops: []
W20260807 03:40:00.199177 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 195385us
I20260807 03:40:00.150687 23411 raft_consensus.cc:482] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:40:00.193544 18248 leader_election.cc:302] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:40:00.173669 23133 raft_consensus.cc:2781] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [term 13 FOLLOWER]: Leader election won for term 13
I20260807 03:40:00.196017 23269 raft_consensus.cc:482] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:40:00.188647 23316 raft_consensus.cc:2726] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:40:00.184850 23021 raft_consensus.cc:482] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 5c9b5062bdfa46af9396f1a85025a331)
I20260807 03:40:00.182930 19289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:40:00.188704 23409 raft_consensus.cc:2726] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260807 03:40:00.107694 22441 raft_consensus.cc:504] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:00.189694 23264 raft_consensus.cc:3032] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 8 LEADER]: Stepping down as leader of term 8
I20260807 03:40:00.109275 23010 consensus_queue.cc:237] T 7b26fcaa55344cf98fc76dd083788d00 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:00.189860 23041 raft_consensus.cc:686] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [term 10 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:40:00.188498 22984 consensus_queue.cc:1035] T 60437eed8852484c88e84993e376670e P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
I20260807 03:40:00.196393 23351 consensus_queue.cc:1035] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
I20260807 03:40:00.232506 23133 raft_consensus.cc:686] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [term 13 LEADER]: Becoming Leader. State: Replica: 5c9b5062bdfa46af9396f1a85025a331, State: Running, Role: LEADER
W20260807 03:40:00.228514 23248 consensus_meta.cc:287] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 0.510s user 0.010s sys 0.002s
W20260807 03:40:00.234366 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 74421.5us
W20260807 03:40:00.250696 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 61508us
I20260807 03:40:00.227415 23021 raft_consensus.cc:3037] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:40:00.223486 23013 leader_election.cc:288] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:40:00.197983 23346 raft_consensus.cc:504] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:00.229528 23243 raft_consensus.cc:482] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:40:00.230696 23264 raft_consensus.cc:729] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:40:00.253167 23153 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:59906 (request call id 221) took 5955 ms. Trace:
I20260807 03:40:00.277063 18193 raft_consensus.cc:1226] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 LEADER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 8. Current term is 9. Ops: []
I20260807 03:40:00.235486 23409 raft_consensus.cc:2781] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:40:00.232729 19289 raft_consensus.cc:2454] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 4.
I20260807 03:40:00.272729 18298 raft_consensus.cc:3037] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Advancing to term 10
I20260807 03:40:00.294900 23243 raft_consensus.cc:504] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:00.301721 23243 leader_election.cc:288] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:40:00.301937 23346 leader_election.cc:288] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:40:00.302690 23423 raft_consensus.cc:482] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:40:00.303179 23423 raft_consensus.cc:504] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:00.219412 23411 raft_consensus.cc:504] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:00.295099 22881 consensus_queue.cc:237] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:00.305256 23423 leader_election.cc:288] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:40:00.305979 23416 consensus_queue.cc:1035] T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.117s
I20260807 03:40:00.306612 23411 leader_election.cc:288] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:40:00.235041 23316 raft_consensus.cc:2726] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 03:40:00.223268 22862 consensus_queue.cc:1046] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.048s
I20260807 03:40:00.272643 19287 raft_consensus.cc:1226] T 6b66966247c147308a4b8488557e0bdf P 5c9b5062bdfa46af9396f1a85025a331 [term 10 LEADER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 9. Current term is 10. Ops: []
I20260807 03:40:00.229382 22762 consensus_queue.cc:1046] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.048s
W20260807 03:40:00.357458 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 136574us
I20260807 03:40:00.364284 18246 leader_election.cc:302] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:40:00.234650 23269 raft_consensus.cc:504] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:00.309577 23409 raft_consensus.cc:482] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:40:00.278112 23248 consensus_queue.cc:1035] T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.089s
W20260807 03:40:00.378237 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 121349us
W20260807 03:40:00.381103 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 194381us
W20260807 03:40:00.382850 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 189672us
I20260807 03:40:00.250243 23133 consensus_queue.cc:237] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:40:00.365967 23416 raft_consensus.cc:482] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:40:00.308236 22896 consensus_queue.cc:260] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:00.372772 23409 raft_consensus.cc:3037] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:40:00.272912 22980 consensus_queue.cc:260] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:00.379101 22762 consensus_queue.cc:1225] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer 3c4bceb836db4536a0f255fdf86abc02 log is divergent from this leader: its last log entry 10.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260807 03:40:00.379616 23316 raft_consensus.cc:2781] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:40:00.352421 23411 consensus_queue.cc:1035] T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20260807 03:40:00.384500 23354 raft_consensus.cc:482] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:40:00.391285 18027 leader_election.cc:302] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:40:00.397315 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:40:00.392866 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
W20260807 03:40:00.398134 23414 log.cc:924] Time spent T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.169s user 0.000s sys 0.001s
I20260807 03:40:00.431851 18305 raft_consensus.cc:2454] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 8.
I20260807 03:40:00.437390 18293 raft_consensus.cc:2454] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 6.
I20260807 03:40:00.410864 23316 raft_consensus.cc:482] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:40:00.399194 23134 consensus_queue.cc:260] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:00.399434 23346 consensus_queue.cc:1035] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20260807 03:40:00.434630 22980 raft_consensus.cc:3037] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Advancing to term 10
I20260807 03:40:00.447860 18298 raft_consensus.cc:2454] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 10.
I20260807 03:40:00.431586 22896 raft_consensus.cc:3037] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:40:00.431828 22762 consensus_queue.cc:1046] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20260807 03:40:00.393887 23264 consensus_queue.cc:260] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:00.496641 23264 raft_consensus.cc:3037] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:40:00.498413 23269 leader_election.cc:288] T 60437eed8852484c88e84993e376670e P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:40:00.500298 23436 consensus_queue.cc:1159] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" responder_term: 9 status { last_received { term: 8 index: 5 } last_committed_idx: 5 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 8. Current term is 9. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260807 03:40:00.394850 23416 raft_consensus.cc:504] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:00.503185 23416 leader_election.cc:288] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:40:00.502895 23343 raft_consensus.cc:504] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:00.412684 23351 consensus_queue.cc:1046] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.035s
I20260807 03:40:00.311354 23153 rpcz_store.cc:276] 0807 03:39:54.291327 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:54.303797 (+ 12470us) service_pool.cc:227] Handling call
0807 03:39:54.828276 (+524479us) ts_tablet_manager.cc:928] T d416e1015a2644958b0c8f9147d403a2 P 5c9b5062bdfa46af9396f1a85025a331: Initiating tablet copy from peer 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
0807 03:39:59.391060 (+4562784us) tablet_copy_client.cc:361] Tablet copy session begun
0807 03:39:59.888224 (+497164us) tablet_metadata.cc:641] Metadata flushed
0807 03:39:59.888619 (+ 395us) tablet_copy_client.cc:490] Wrote new tablet metadata
0807 03:40:00.153675 (+265056us) tablet_copy_client.cc:411] Wrote new consensus metadata
0807 03:40:00.162496 (+ 8821us) tablet_replica.cc:205] Creating consensus instance
0807 03:40:00.198057 (+ 35561us) ts_tablet_manager.cc:983] Replica registered.
0807 03:40:00.223170 (+ 25113us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":8806,"tablet-copy.queue_time_us":380722,"thread_start_us":1277342,"threads_started":2}
W20260807 03:40:00.476742 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 91460.4us
W20260807 03:40:00.478256 23419 log.cc:924] Time spent T bb66518c65e4495e881ce860253d0801 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.129s user 0.000s sys 0.001s
W20260807 03:40:00.416522 23318 consensus_queue.cc:1159] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" responder_term: 10 status { last_received { term: 8 index: 6 } last_committed_idx: 6 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 9. Current term is 10. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260807 03:40:00.488821 23271 raft_consensus.cc:504] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:40:00.435294 22881 raft_consensus.cc:482] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:40:00.435441 23354 raft_consensus.cc:504] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:40:00.440346 23284 raft_consensus.cc:504] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:00.385927 23041 consensus_queue.cc:237] T c1476cce1786402ab984d773b26acb1b P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:40:00.471633 23410 log.cc:924] Time spent T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.114s user 0.000s sys 0.001s
I20260807 03:40:00.499982 23269 raft_consensus.cc:2726] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260807 03:40:00.447131 23316 raft_consensus.cc:3037] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
W20260807 03:40:00.534086 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 152337us
I20260807 03:40:00.460019 22441 leader_election.cc:288] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:40:00.508594 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 75984.2us
W20260807 03:40:00.532645 23150 consensus_meta.cc:287] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.518s user 0.012s sys 0.002s
I20260807 03:40:00.527022 23167 raft_consensus.cc:504] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:40:00.558223 23261 raft_consensus.cc:2726] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260807 03:40:00.558674 23167 leader_election.cc:288] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 9 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:40:00.559144 19289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331"
I20260807 03:40:00.560032 19289 raft_consensus.cc:2379] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 9: Already voted for candidate 5c9b5062bdfa46af9396f1a85025a331 in this term.
I20260807 03:40:00.560966 19297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:40:00.562690 18073 raft_consensus.cc:1261] T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260807 03:40:00.563640 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:40:00.564325 18080 raft_consensus.cc:2454] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 6.
I20260807 03:40:00.565864 22847 raft_consensus.cc:3032] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 03:40:00.506464 23145 raft_consensus.cc:482] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:40:00.549669 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:40:00.550802 18072 raft_consensus.cc:1261] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260807 03:40:00.551823 23432 consensus_queue.cc:1035] T 4ae0265e0707432980672f7cc955c571 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260807 03:40:00.552440 23271 leader_election.cc:288] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 election: Requested vote from peers 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:40:00.553074 23354 leader_election.cc:288] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:40:00.559315 23134 raft_consensus.cc:3037] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:40:00.548497 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:40:00.542858 23021 raft_consensus.cc:504] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:40:00.566320 22847 raft_consensus.cc:729] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:40:00.577978 18293 raft_consensus.cc:3032] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 9 LEADER]: Stepping down as leader of term 9
I20260807 03:40:00.576113 18070 raft_consensus.cc:2454] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 7.
I20260807 03:40:00.581319 18293 raft_consensus.cc:729] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:40:00.586127 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:40:00.587209 18073 raft_consensus.cc:2396] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260807 03:40:00.569412 23343 leader_election.cc:288] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:40:00.607390 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:40:00.610211 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:40:00.610244 23150 consensus_queue.cc:1046] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 03:40:00.610895 18305 raft_consensus.cc:2454] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 8.
I20260807 03:40:00.611449 23150 raft_consensus.cc:3032] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 03:40:00.611867 23150 raft_consensus.cc:729] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:40:00.575347 23045 raft_consensus.cc:482] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:40:00.616339 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:40:00.619505 18141 leader_election.cc:302] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:40:00.588811 23271 raft_consensus.cc:482] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:40:00.622423 18191 raft_consensus.cc:1226] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 14 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 13. Current term is 14. Ops: []
I20260807 03:40:00.613538 23442 raft_consensus.cc:482] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:40:00.629096 23442 raft_consensus.cc:504] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:00.629688 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:40:00.630645 18183 raft_consensus.cc:2379] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 5c9b5062bdfa46af9396f1a85025a331 in current term 9: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:40:00.631349 23442 leader_election.cc:288] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:40:00.633049 18191 raft_consensus.cc:1261] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Refusing update from remote peer 5c9b5062bdfa46af9396f1a85025a331: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260807 03:40:00.634218 18184 raft_consensus.cc:1226] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 11 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 10. Current term is 11. Ops: []
I20260807 03:40:00.635643 22847 consensus_queue.cc:260] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:00.637123 22847 raft_consensus.cc:3037] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:40:00.640511 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c5f72c354324b78146f4ed947cad" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:40:00.642038 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:40:00.642727 18312 raft_consensus.cc:3037] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:40:00.643122 18140 leader_election.cc:302] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:40:00.644251 23270 raft_consensus.cc:482] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:40:00.584650 23145 raft_consensus.cc:504] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:00.645498 23001 consensus_queue.cc:1035] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260807 03:40:00.647140 23001 raft_consensus.cc:2781] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 03:40:00.647508 23001 raft_consensus.cc:482] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:40:00.590292 22984 raft_consensus.cc:482] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:40:00.615057 18293 consensus_queue.cc:260] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:00.648841 18185 raft_consensus.cc:2454] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 10.
I20260807 03:40:00.674892 18081 raft_consensus.cc:1261] T 7285c5f72c354324b78146f4ed947cad P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260807 03:40:00.676216 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:40:00.680014 18193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:40:00.597798 23354 raft_consensus.cc:482] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:40:00.657321 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:40:00.682237 18311 raft_consensus.cc:3037] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:40:00.631982 18183 raft_consensus.cc:1226] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 LEADER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 5. Current term is 6. Ops: []
I20260807 03:40:00.651659 18140 leader_election.cc:302] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:40:00.692106 18075 raft_consensus.cc:1261] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260807 03:40:00.695103 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:40:00.569921 22881 raft_consensus.cc:504] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:00.593910 23153 tablet_copy_client.cc:791] T d416e1015a2644958b0c8f9147d403a2 P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Starting download of 0 data blocks...
I20260807 03:40:00.658030 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
W20260807 03:40:00.720830 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 69252.1us
I20260807 03:40:00.724444 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349190ae153f46c9a99c29df20734d3f" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:40:00.727706 18140 leader_election.cc:302] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:40:00.716876 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:40:00.728978 18070 raft_consensus.cc:1261] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260807 03:40:00.650954 23411 raft_consensus.cc:2781] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 03:40:00.647887 23001 raft_consensus.cc:3037] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:40:00.730273 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d6cc12bed1425985f0504451c5e057" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:40:00.735741 18293 raft_consensus.cc:3037] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Advancing to term 10
I20260807 03:40:00.729374 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:40:00.644877 23270 raft_consensus.cc:504] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:00.732034 18081 raft_consensus.cc:2454] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 9.
I20260807 03:40:00.733006 18073 raft_consensus.cc:2454] T 349190ae153f46c9a99c29df20734d3f P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 7.
I20260807 03:40:00.737381 18082 raft_consensus.cc:1261] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Refusing update from remote peer 66a36e1ac8e945e893668e282d394c49: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 03:40:00.741093 18075 raft_consensus.cc:2379] T e8d6cc12bed1425985f0504451c5e057 P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 10: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:40:00.750738 18072 raft_consensus.cc:1226] T 28ccf467b7d24422ba092003f6b06616 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 7. Current term is 8. Ops: []
I20260807 03:40:00.758662 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d26ed9a1e340ce8c2d127ac7fbf7f7" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:40:00.647786 23045 raft_consensus.cc:504] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:00.720679 23284 leader_election.cc:288] T c702701ec4504375977f142ffc7f0be0 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:40:00.732728 23261 raft_consensus.cc:2781] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 03:40:00.765177 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "5c9b5062bdfa46af9396f1a85025a331" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:40:00.652081 23346 consensus_queue.cc:1035] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.090s
I20260807 03:40:00.736343 23411 raft_consensus.cc:482] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:40:00.695266 23354 raft_consensus.cc:504] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:40:00.718748 22984 raft_consensus.cc:504] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:00.779452 18311 raft_consensus.cc:2454] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 9.
I20260807 03:40:00.656920 23150 consensus_queue.cc:260] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:00.730172 23271 raft_consensus.cc:504] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:00.772966 23261 raft_consensus.cc:482] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:40:00.788401 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:40:00.773205 23153 tablet_copy_client.cc:655] T d416e1015a2644958b0c8f9147d403a2 P 5c9b5062bdfa46af9396f1a85025a331: tablet copy: Starting download of 1 WAL segments...
I20260807 03:40:00.780375 23411 raft_consensus.cc:3037] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:40:00.800336 18297 raft_consensus.cc:2454] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 9.
I20260807 03:40:00.804394 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:40:00.804723 18073 raft_consensus.cc:1261] T 79cf89cd640d48bcbbf376c2d01e79d4 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
W20260807 03:40:00.830121 23451 log.cc:924] Time spent T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.297s user 0.000s sys 0.001s
W20260807 03:40:00.831707 23394 log.cc:924] Time spent T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.057s user 0.001s sys 0.000s
I20260807 03:40:00.833077 23150 raft_consensus.cc:3037] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:40:00.904644 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:40:00.905375 18069 raft_consensus.cc:2454] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 11.
I20260807 03:40:00.796217 23261 raft_consensus.cc:3037] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:40:00.910357 23316 raft_consensus.cc:504] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:40:00.781554 23145 leader_election.cc:288] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:40:00.912685 23145 raft_consensus.cc:482] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:40:00.913657 23013 raft_consensus.cc:3032] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Stepping down as leader of term 5
I20260807 03:40:00.914048 23013 raft_consensus.cc:729] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:40:00.848694 18298 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37190 (request call id 200) took 1331 ms. Trace:
I20260807 03:40:00.849062 23132 raft_consensus.cc:3032] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 10 LEADER]: Stepping down as leader of term 10
I20260807 03:40:00.915059 23132 raft_consensus.cc:729] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:40:00.850415 23271 leader_election.cc:288] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:40:00.850894 23270 leader_election.cc:288] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:40:00.794622 23010 raft_consensus.cc:482] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5c9b5062bdfa46af9396f1a85025a331)
W20260807 03:40:00.851626 23456 log.cc:924] Time spent T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20260807 03:40:00.830718 23430 raft_consensus.cc:482] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:40:00.831148 23374 consensus_queue.cc:1035] T 6b66966247c147308a4b8488557e0bdf P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.102s
W20260807 03:40:00.848577 23402 log.cc:924] Time spent T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260807 03:40:00.854036 22984 leader_election.cc:288] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:40:00.855020 22881 leader_election.cc:288] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:40:00.835593 23468 raft_consensus.cc:482] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:40:00.834479 23439 raft_consensus.cc:2726] T 7285c5f72c354324b78146f4ed947cad P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260807 03:40:00.856693 23045 leader_election.cc:288] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:40:00.857663 18293 raft_consensus.cc:1261] T 2371c1d6174640aa9a9ee19732634a88 P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260807 03:40:00.855875 23001 raft_consensus.cc:504] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:00.859140 18311 raft_consensus.cc:1261] T 9a34af1cebc14f92bad043893aa1f964 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260807 03:40:00.866209 18312 raft_consensus.cc:2454] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c9b5062bdfa46af9396f1a85025a331 in term 6.
W20260807 03:40:00.868163 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 50207.8us
I20260807 03:40:00.808302 18072 raft_consensus.cc:2379] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 7: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:40:00.901249 23354 leader_election.cc:288] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:40:00.903831 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58a8dc36ec94e0c8877bdf0f901598a" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:40:00.875581 23442 consensus_queue.cc:1035] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20260807 03:40:00.913328 23145 raft_consensus.cc:504] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:40:00.797042 23409 raft_consensus.cc:504] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:40:00.846406 23397 log.cc:924] Time spent T 0890f92b53f04149998f1f1c4e7ba60d P 66a36e1ac8e945e893668e282d394c49: Append to log took a long time: real 0.868s user 0.002s sys 0.002s
I20260807 03:40:00.916775 18193 raft_consensus.cc:2396] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260807 03:40:00.914517 18298 rpcz_store.cc:276] 0807 03:39:59.517562 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:59.517721 (+ 159us) service_pool.cc:227] Handling call
0807 03:39:59.873304 (+355583us) spinlock_profiling.cc:243] Waited 2.61 ms on lock 0x7b540012cb20. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe992155951 00007fe9921b6321 00007fe9921b7b4e 00007fe9921c3228 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2
0807 03:40:00.266991 (+393687us) spinlock_profiling.cc:243] Waited 158 ms on lock 0x7b58008cf0b0. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe99213af63 00007fe99219eaba 00007fe9921b63af 00007fe9921b7b4e 00007fe9921c3228 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c
0807 03:40:00.848470 (+581479us) spinlock_profiling.cc:243] Waited 230 ms on lock 0x7b54001c3754. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921c32ae 00007fe997100c27 00007fe9907e8c04 00007fe9907e8b92 00007fe9907e8ac1 00007fe9907e8a3c 00007fe9907e7cf2 00007fe9905e994c 00007fe9905e8d86 00007fe9905e872f
0807 03:40:00.848607 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":898636160}
I20260807 03:40:00.939137 18180 raft_consensus.cc:2454] T e8d6cc12bed1425985f0504451c5e057 P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 10.
I20260807 03:40:00.914909 23013 consensus_queue.cc:260] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
W20260807 03:40:00.984586 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 444983us
W20260807 03:40:00.971879 23427 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 10 current term: 10 is higher.
I20260807 03:40:00.985072 23316 leader_election.cc:288] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:40:00.951666 23010 raft_consensus.cc:504] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:40:00.988330 23132 consensus_queue.cc:260] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:01.070223 23010 leader_election.cc:288] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:40:01.067512 18027 leader_election.cc:302] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:40:01.068229 23134 raft_consensus.cc:482] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:40:01.071571 23134 raft_consensus.cc:504] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:40:01.073280 18027 leader_election.cc:302] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 66a36e1ac8e945e893668e282d394c49
I20260807 03:40:01.073683 23134 leader_election.cc:288] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:40:01.069147 18309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57268 (request call id 2197) took 1267 ms. Trace:
I20260807 03:40:01.074245 23470 raft_consensus.cc:2741] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260807 03:40:01.074911 23470 consensus_queue.cc:1046] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 03:40:00.951943 23430 raft_consensus.cc:504] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:01.074267 18309 rpcz_store.cc:276] 0807 03:39:59.801414 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:39:59.802915 (+ 1501us) service_pool.cc:227] Handling call
0807 03:39:59.814415 (+ 11500us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:39:59.814782 (+ 367us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:39:59.814813 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:39:59.815218 (+ 405us) log.cc:842] Serialized 26 byte log entry
0807 03:39:59.857271 (+ 42053us) raft_consensus.cc:1631] Marking committed up to 5
0807 03:39:59.857329 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:39:59.857432 (+ 103us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:40:00.845656 (+988224us) spinlock_profiling.cc:243] Waited 1.17 ms on lock 0x7b44000232cc. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f1a9a95 00007fe98f1aa4b2 00007fe991e52511 00007fe991e4de1a 00007fe991e41bee 00007fe991e40f36 00007fe991e40b58 00007fe991e4a551 00007fe991e4a509 00007fe991e4a499
0807 03:40:01.068830 (+223174us) raft_consensus.cc:1671] finished
0807 03:40:01.068853 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:40:01.069069 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":99,"raft.run_cpu_time_us":74,"raft.run_wall_time_us":75,"spinlock_wait_cycles":2700160,"thread_start_us":41666,"threads_started":1,"wal-append.queue_time_us":117617}
I20260807 03:40:00.992828 18073 raft_consensus.cc:2379] T b58a8dc36ec94e0c8877bdf0f901598a P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 in current term 5: Already voted for candidate d9716e2bf06442f5b3b0e81087c53c4d in this term.
I20260807 03:40:01.077561 23457 raft_consensus.cc:3032] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 LEADER]: Stepping down as leader of term 7
I20260807 03:40:01.077907 23457 raft_consensus.cc:729] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:40:01.078267 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:40:01.078846 18073 raft_consensus.cc:2454] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 7.
I20260807 03:40:01.078722 23457 consensus_queue.cc:260] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:01.079715 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:40:01.080693 18027 leader_election.cc:302] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:40:01.080811 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:40:01.082676 23409 leader_election.cc:288] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:40:01.083014 18027 leader_election.cc:398] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 election: Vote denied by peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057) with higher term. Message: Invalid argument: T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 9. Current term is 10.
I20260807 03:40:01.083487 18027 leader_election.cc:401] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20260807 03:40:01.075995 23470 raft_consensus.cc:2781] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election won for term 5
I20260807 03:40:01.084282 23470 raft_consensus.cc:686] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:40:01.085386 23470 consensus_queue.cc:237] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:01.080009 23457 raft_consensus.cc:3037] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:40:01.068171 23013 raft_consensus.cc:3037] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Advancing to term 6
I20260807 03:40:00.995580 18140 leader_election.cc:302] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
W20260807 03:40:01.098502 18140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 103017us
I20260807 03:40:01.101545 18140 leader_election.cc:302] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:40:00.999820 23248 raft_consensus.cc:482] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:40:00.996595 23270 raft_consensus.cc:482] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:40:01.108520 23409 raft_consensus.cc:2781] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260807 03:40:01.000242 23411 raft_consensus.cc:504] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:01.114784 23271 raft_consensus.cc:2781] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 03:40:01.111891 23010 raft_consensus.cc:2726] T 13bfb55b1d7f43dc9fb01c519c175ccf P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260807 03:40:01.180932 23475 log.cc:924] Time spent T 8c98c1d422654739af06653a512f7499 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.190s user 0.000s sys 0.001s
I20260807 03:40:01.181110 23001 leader_election.cc:288] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:40:01.167239 23271 raft_consensus.cc:482] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:40:01.184120 23271 raft_consensus.cc:3037] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:40:01.185163 23261 raft_consensus.cc:504] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:01.190110 18310 raft_consensus.cc:2454] T b58a8dc36ec94e0c8877bdf0f901598a P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 5.
I20260807 03:40:01.190431 23442 raft_consensus.cc:482] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 03:40:01.191421 22847 consensus_meta.cc:287] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.554s user 0.012s sys 0.005s
I20260807 03:40:01.191071 23442 raft_consensus.cc:504] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:01.164925 23409 raft_consensus.cc:482] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:40:01.115304 23248 raft_consensus.cc:504] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:01.192116 23409 raft_consensus.cc:3037] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 11 FOLLOWER]: Advancing to term 12
I20260807 03:40:01.193502 23248 leader_election.cc:288] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:40:01.195735 23132 raft_consensus.cc:3037] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Advancing to term 11
I20260807 03:40:01.200984 18293 raft_consensus.cc:1261] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260807 03:40:01.204607 23261 leader_election.cc:288] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:40:01.205674 23411 leader_election.cc:288] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:40:01.174551 23201 raft_consensus.cc:482] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:40:01.135900 23270 raft_consensus.cc:504] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:00.988859 23468 raft_consensus.cc:504] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:01.193535 23442 leader_election.cc:288] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:40:01.236727 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:40:01.243973 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:40:01.245123 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:40:01.245744 18079 raft_consensus.cc:2454] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 9.
I20260807 03:40:01.246626 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:40:01.247289 18079 raft_consensus.cc:3037] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:40:01.254113 18140 leader_election.cc:302] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:40:01.224426 23430 leader_election.cc:288] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:40:01.255366 23480 raft_consensus.cc:482] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:40:01.260542 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:40:01.276170 23430 raft_consensus.cc:2781] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260807 03:40:01.276747 23430 raft_consensus.cc:482] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:40:01.277205 23430 raft_consensus.cc:3037] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Advancing to term 10
I20260807 03:40:01.251176 23041 raft_consensus.cc:2726] T 4ae0265e0707432980672f7cc955c571 P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057) with higher term. Message: Invalid argument: T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d for earlier term 9. Current term is 10.
I20260807 03:40:01.238996 23201 raft_consensus.cc:504] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:01.309871 18188 raft_consensus.cc:3037] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 12 FOLLOWER]: Advancing to term 13
I20260807 03:40:01.308882 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:40:01.315696 18075 raft_consensus.cc:2454] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 10.
I20260807 03:40:01.316105 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:40:01.316722 18069 raft_consensus.cc:3037] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:40:01.318696 18140 leader_election.cc:302] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:40:01.281702 23145 leader_election.cc:288] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:40:01.322793 19297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:40:01.322958 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:40:01.323599 19297 raft_consensus.cc:2454] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 4.
I20260807 03:40:01.324316 23442 raft_consensus.cc:2781] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260807 03:40:01.337002 18075 raft_consensus.cc:1261] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
W20260807 03:40:01.338985 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 256162us
I20260807 03:40:01.339931 23145 consensus_queue.cc:1035] T 33252bff379e4392a9a815ecdbc447a7 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 03:40:01.339977 18025 leader_election.cc:302] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:40:01.342494 23508 raft_consensus.cc:2781] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 03:40:01.342911 23508 raft_consensus.cc:482] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 5c9b5062bdfa46af9396f1a85025a331)
I20260807 03:40:01.343266 23508 raft_consensus.cc:3037] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Advancing to term 4
I20260807 03:40:01.348870 23270 leader_election.cc:288] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:40:01.349912 18079 raft_consensus.cc:2454] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 8.
I20260807 03:40:01.350628 23468 leader_election.cc:288] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:40:01.353312 18079 raft_consensus.cc:1261] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260807 03:40:01.351955 18070 raft_consensus.cc:1261] T 6129789267024fac8a337070c10a472a P d9716e2bf06442f5b3b0e81087c53c4d [term 13 FOLLOWER]: Refusing update from remote peer 5c9b5062bdfa46af9396f1a85025a331: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260807 03:40:01.355393 23430 raft_consensus.cc:504] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:01.360776 18188 raft_consensus.cc:1261] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 13 FOLLOWER]: Refusing update from remote peer 5c9b5062bdfa46af9396f1a85025a331: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260807 03:40:01.363077 18140 leader_election.cc:302] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:40:01.279824 23480 raft_consensus.cc:504] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:40:01.370673 23508 raft_consensus.cc:504] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:40:01.327576 23271 raft_consensus.cc:504] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:01.374639 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:40:01.376950 23480 leader_election.cc:288] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:40:01.337891 23442 raft_consensus.cc:482] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:40:01.387660 18027 leader_election.cc:302] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:40:01.389410 19289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:40:01.391376 18073 raft_consensus.cc:2454] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 8.
I20260807 03:40:01.422456 19297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:40:01.397414 18069 raft_consensus.cc:2454] T b2d369700aef4621b22ba058956d0894 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 7.
I20260807 03:40:01.374678 23201 leader_election.cc:288] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:40:01.442083 19297 raft_consensus.cc:2454] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 8.
I20260807 03:40:01.453397 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:40:01.454174 18140 leader_election.cc:302] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:40:01.454809 19291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15fdfae36e94837adea3c6b66864b0e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:40:01.461985 23037 consensus_queue.cc:1035] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260807 03:40:01.462376 23417 raft_consensus.cc:482] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:40:01.462853 23417 raft_consensus.cc:504] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:40:01.463536 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 75955.2us
I20260807 03:40:01.465163 23417 leader_election.cc:288] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:40:01.466127 23421 raft_consensus.cc:2781] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 03:40:01.466815 23421 raft_consensus.cc:686] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:40:01.468195 23021 leader_election.cc:288] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 election: Requested vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:40:01.467922 23421 consensus_queue.cc:237] T b2d369700aef4621b22ba058956d0894 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:01.422626 22476 raft_consensus.cc:2781] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader election won for term 8
I20260807 03:40:01.434369 23442 raft_consensus.cc:3037] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Advancing to term 11
I20260807 03:40:01.465095 18027 leader_election.cc:302] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:40:01.492833 18138 leader_election.cc:302] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 5c9b5062bdfa46af9396f1a85025a331; no voters:
I20260807 03:40:01.501375 18188 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "13bfb55b1d7f43dc9fb01c519c175ccf"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
mode: GRACEFUL
new_leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:34758
I20260807 03:40:01.501994 18081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:40:01.456982 23480 raft_consensus.cc:2726] T 922e1d3cde8f476ea7780363c6b8693c P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:40:01.502054 18188 raft_consensus.cc:595] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Received request to transfer leadership to TS d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:40:01.503554 18188 raft_consensus.cc:603] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 03:40:01.391458 23409 raft_consensus.cc:504] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:01.506731 23409 leader_election.cc:288] T 8a9b0210bce148c4a06eef3019b593c6 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 12 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:40:01.454912 23426 raft_consensus.cc:482] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:40:01.518960 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ca0223f47b46fd955704eba5259370" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:40:01.517781 18138 leader_election.cc:302] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:40:01.519650 18069 raft_consensus.cc:2454] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 3.
W20260807 03:40:01.520495 19240 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 153371us
I20260807 03:40:01.459400 23468 consensus_queue.cc:1035] T e5ea193a7604411b81684ccff437bc0a P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.060s
I20260807 03:40:01.496704 22476 raft_consensus.cc:686] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
I20260807 03:40:01.513592 23417 raft_consensus.cc:2781] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260807 03:40:01.522769 19240 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.15.198.131:33021 timed out after 2.124s (SENT) [suppressed 23 similar messages]
W20260807 03:40:01.538149 19239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 184356us
I20260807 03:40:01.545584 18082 raft_consensus.cc:1261] T 6b66966247c147308a4b8488557e0bdf P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Refusing update from remote peer 5c9b5062bdfa46af9396f1a85025a331: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260807 03:40:01.531414 23013 raft_consensus.cc:3037] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Advancing to term 10
I20260807 03:40:01.531623 23270 raft_consensus.cc:2726] T b15fdfae36e94837adea3c6b66864b0e P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 03:40:01.546456 18140 leader_election.cc:302] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
W20260807 03:40:01.556233 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 91226.4us
I20260807 03:40:01.545209 23417 raft_consensus.cc:482] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:40:01.564461 23417 raft_consensus.cc:3037] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Advancing to term 10
I20260807 03:40:01.568655 18027 leader_election.cc:302] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:40:01.569804 23269 raft_consensus.cc:2726] T 33252bff379e4392a9a815ecdbc447a7 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260807 03:40:01.570901 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 919577us
I20260807 03:40:01.571118 23411 raft_consensus.cc:2781] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:40:01.571533 23411 raft_consensus.cc:482] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:40:01.571862 23411 raft_consensus.cc:3037] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
W20260807 03:40:01.580710 23511 log.cc:924] Time spent T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20260807 03:40:01.582098 23261 raft_consensus.cc:482] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:40:01.538471 19287 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59906 (request call id 244) took 1434 ms. Trace:
I20260807 03:40:01.572765 18249 leader_election.cc:302] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:40:01.585937 18071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33268 (request call id 2126) took 1035 ms. Trace:
I20260807 03:40:01.571801 18027 leader_election.cc:302] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:40:01.584483 18140 leader_election.cc:302] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:40:01.594558 18141 leader_election.cc:302] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: 5c9b5062bdfa46af9396f1a85025a331
W20260807 03:40:01.579447 23516 log.cc:924] Time spent T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.079s user 0.000s sys 0.002s
W20260807 03:40:01.572830 23519 log.cc:924] Time spent T bb66518c65e4495e881ce860253d0801 P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.056s user 0.000s sys 0.001s
I20260807 03:40:01.536029 23426 raft_consensus.cc:504] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:01.600980 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:40:01.611373 18079 raft_consensus.cc:3037] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:40:01.601207 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:40:01.612170 18070 raft_consensus.cc:3037] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Advancing to term 10
I20260807 03:40:01.602707 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:40:01.613539 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:40:01.614075 18073 raft_consensus.cc:3037] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 11 FOLLOWER]: Advancing to term 12
I20260807 03:40:01.616842 18141 leader_election.cc:302] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:40:01.578238 23430 leader_election.cc:288] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:40:01.614209 18310 raft_consensus.cc:2454] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 7.
I20260807 03:40:01.620886 18298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:40:01.621996 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd704177fca4d3bbd1b9d39a5f3ec01" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:40:01.628958 18295 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5"
dest_uuid: "66a36e1ac8e945e893668e282d394c49"
mode: GRACEFUL
new_leader_uuid: "3c4bceb836db4536a0f255fdf86abc02"
from {username='slave'} at 127.0.0.1:58736
I20260807 03:40:01.629734 18295 raft_consensus.cc:595] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Received request to transfer leadership to TS 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:40:01.630113 18295 raft_consensus.cc:603] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 03:40:01.563560 22476 consensus_queue.cc:237] T 6399a0e5977a46c8b09c11f11af795e8 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:01.623342 18249 leader_election.cc:302] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters:
W20260807 03:40:01.621544 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 49819.7us
I20260807 03:40:01.664620 18027 leader_election.cc:302] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
W20260807 03:40:01.667374 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 50630.5us
I20260807 03:40:01.668187 19291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331"
I20260807 03:40:01.668879 19291 raft_consensus.cc:3037] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:40:01.669010 23510 consensus_queue.cc:1035] T 6129789267024fac8a337070c10a472a P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.114s
I20260807 03:40:01.671427 19297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331"
I20260807 03:40:01.672405 19297 raft_consensus.cc:2354] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 8. Current term is 9.
I20260807 03:40:01.672509 23457 raft_consensus.cc:2781] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Leader election won for term 6
I20260807 03:40:01.632963 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:40:01.673142 23457 raft_consensus.cc:686] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [term 6 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:40:01.673454 18297 raft_consensus.cc:2454] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 11.
I20260807 03:40:01.674036 23457 consensus_queue.cc:237] T c3de9873c6144d36b33a1a666691869a P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:01.603920 19289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:40:01.654186 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:40:01.600914 23467 raft_consensus.cc:482] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:40:01.609167 23284 raft_consensus.cc:482] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:40:01.691607 18070 raft_consensus.cc:2454] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 10.
I20260807 03:40:01.631618 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:40:01.616248 23248 raft_consensus.cc:2781] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader election won for term 9
W20260807 03:40:01.681766 19240 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 159022us
I20260807 03:40:01.574308 23264 consensus_queue.cc:1046] T 67de6ea5223f4eefa2edd875f9dfd1bc P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.047s
I20260807 03:40:01.621479 23343 raft_consensus.cc:2781] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260807 03:40:01.582808 23261 raft_consensus.cc:504] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:01.678413 18311 raft_consensus.cc:2454] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 9.
I20260807 03:40:01.686164 19297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331"
I20260807 03:40:01.596999 18071 rpcz_store.cc:276] 0807 03:40:00.550359 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:40:00.550528 (+ 169us) service_pool.cc:227] Handling call
0807 03:40:00.550976 (+ 448us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:40:00.779856 (+228880us) raft_consensus.cc:1520] Early marking committed up to index 5
0807 03:40:01.092939 (+313083us) log.cc:842] Serialized 14 byte log entry
0807 03:40:01.313373 (+220434us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:40:01.329752 (+ 16379us) log.cc:842] Serialized 26 byte log entry
0807 03:40:01.330090 (+ 338us) raft_consensus.cc:1631] Marking committed up to 6
0807 03:40:01.331630 (+ 1540us) log.cc:842] Serialized 14 byte log entry
0807 03:40:01.332041 (+ 411us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:40:01.332197 (+ 156us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:40:01.585671 (+253474us) raft_consensus.cc:1671] finished
0807 03:40:01.585698 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:40:01.585866 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13345,"prepare.queue_time_us":512458,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":66,"raft.queue_time_us":70170,"raft.run_cpu_time_us":300,"raft.run_wall_time_us":58896,"thread_start_us":708105,"threads_started":3,"wal-append.queue_time_us":333571}
I20260807 03:40:01.626029 23432 raft_consensus.cc:482] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5c9b5062bdfa46af9396f1a85025a331)
I20260807 03:40:01.628031 23271 leader_election.cc:288] T 1f5db05c008b404288067417c3f100d2 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:40:01.657238 23508 leader_election.cc:288] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 4 election: Requested vote from peers 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:40:01.653128 23480 raft_consensus.cc:2781] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 11 FOLLOWER]: Leader election won for term 11
I20260807 03:40:01.663571 23422 raft_consensus.cc:482] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:40:01.653213 23409 raft_consensus.cc:2781] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 03:40:01.719038 18081 raft_consensus.cc:1226] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 11 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 10. Current term is 11. Ops: []
I20260807 03:40:01.595136 19287 rpcz_store.cc:276] 0807 03:40:00.100657 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:40:00.102618 (+ 1961us) service_pool.cc:227] Handling call
0807 03:40:00.152923 (+ 50305us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:40:00.366260 (+213337us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:40:01.514002 (+1147742us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b54006cfa54. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe992144471 00007fe9921bff51 00007fe9921be99e 00007fe9971002e9 00007fe9907e6684 00007fe9907e6612 00007fe9907e6541 00007fe9907e64bc 00007fe9907e5772 00007fe9905e994c 00007fe9905e8d86
0807 03:40:01.531467 (+ 17465us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":233588096}
I20260807 03:40:01.716285 18079 raft_consensus.cc:2454] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 8.
I20260807 03:40:01.740387 19240 leader_election.cc:302] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:40:01.724078 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d369700aef4621b22ba058956d0894" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:40:01.715768 23343 raft_consensus.cc:482] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:40:01.723099 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:40:01.620755 22441 consensus_queue.cc:1035] T 2371c1d6174640aa9a9ee19732634a88 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.080s
I20260807 03:40:01.742836 19240 leader_election.cc:302] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:40:01.689654 23207 raft_consensus.cc:482] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:40:01.776342 18073 raft_consensus.cc:2454] T 8a9b0210bce148c4a06eef3019b593c6 P d9716e2bf06442f5b3b0e81087c53c4d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 12.
I20260807 03:40:01.626348 23001 consensus_queue.cc:1035] T 9a34af1cebc14f92bad043893aa1f964 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.087s
I20260807 03:40:01.728647 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071a939ead594d38851e3b554bfc4d24" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:40:01.782778 18082 raft_consensus.cc:3037] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:40:01.733903 18069 raft_consensus.cc:1226] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 7. Current term is 8. Ops: []
I20260807 03:40:01.741524 18297 raft_consensus.cc:3037] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Advancing to term 7
I20260807 03:40:01.735776 23533 consensus_queue.cc:1035] T 6b66966247c147308a4b8488557e0bdf P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.082s
W20260807 03:40:01.722708 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 99461.4us
I20260807 03:40:01.708940 19289 raft_consensus.cc:2454] T 1dbff0b8e97846ea9f567f377b986efc P 5c9b5062bdfa46af9396f1a85025a331 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 8.
I20260807 03:40:01.721446 23150 raft_consensus.cc:482] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:40:01.715646 23243 raft_consensus.cc:482] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:40:01.721738 23010 raft_consensus.cc:482] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:40:01.736184 23430 raft_consensus.cc:482] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:40:01.859992 18310 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4"
dest_uuid: "66a36e1ac8e945e893668e282d394c49"
mode: GRACEFUL
new_leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:58736
I20260807 03:40:01.743582 23480 raft_consensus.cc:686] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [term 11 LEADER]: Becoming Leader. State: Replica: d9716e2bf06442f5b3b0e81087c53c4d, State: Running, Role: LEADER
I20260807 03:40:01.803558 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:40:01.723577 23284 raft_consensus.cc:504] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:40:01.741281 18027 connection.cc:232] RPC call timeout handler was delayed by -1.0227919458411634s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1975s for a response
I20260807 03:40:01.740890 23271 raft_consensus.cc:2781] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 03:40:01.741230 22980 raft_consensus.cc:2781] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Leader election won for term 10
I20260807 03:40:01.741310 23409 raft_consensus.cc:482] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 03:40:01.758386 23448 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.527s user 0.024s sys 0.010s
I20260807 03:40:01.906356 18248 leader_election.cc:302] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:40:01.908931 19291 raft_consensus.cc:2454] T 071a939ead594d38851e3b554bfc4d24 P 5c9b5062bdfa46af9396f1a85025a331 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:40:01.727141 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:40:01.774868 23343 raft_consensus.cc:3037] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Advancing to term 11
I20260807 03:40:01.786010 23167 raft_consensus.cc:2726] T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260807 03:40:01.913187 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:40:01.757249 23421 consensus_queue.cc:1035] T 79cf89cd640d48bcbbf376c2d01e79d4 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.157s
I20260807 03:40:01.913980 23132 consensus_queue.cc:1035] T 1f5db05c008b404288067417c3f100d2 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
W20260807 03:40:01.914808 18027 connection.cc:232] RPC call timeout handler was delayed by -1.0092919459566474s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.199s for a response
I20260807 03:40:01.911768 23534 raft_consensus.cc:2781] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 03:40:01.781873 23201 raft_consensus.cc:482] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 5c9b5062bdfa46af9396f1a85025a331)
I20260807 03:40:01.700168 23467 raft_consensus.cc:504] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:01.924023 18080 raft_consensus.cc:2454] T 303e3bbfb1c74e1686eae36683a96314 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 8.
I20260807 03:40:01.886758 23427 raft_consensus.cc:482] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:40:01.913745 18310 raft_consensus.cc:595] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Received request to transfer leadership to TS d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:40:01.957484 18310 raft_consensus.cc:603] T 79cf89cd640d48bcbbf376c2d01e79d4 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 03:40:01.945942 18082 raft_consensus.cc:2454] T 071a939ead594d38851e3b554bfc4d24 P d9716e2bf06442f5b3b0e81087c53c4d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 5.
I20260807 03:40:01.926111 23409 raft_consensus.cc:3037] T 303e3bbfb1c74e1686eae36683a96314 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:40:01.934212 18311 raft_consensus.cc:2454] T 96140f11b8e74293b333cd7e760c3959 P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 10.
I20260807 03:40:01.727787 23432 raft_consensus.cc:504] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:40:01.942080 18248 leader_election.cc:302] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 5c9b5062bdfa46af9396f1a85025a331, d9716e2bf06442f5b3b0e81087c53c4d
W20260807 03:40:01.926298 23448 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.695s user 0.025s sys 0.010s
I20260807 03:40:01.744656 23422 raft_consensus.cc:504] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:01.913038 22847 consensus_queue.cc:1035] T 91555cf08ec8405c9d002ad0c11f8e5e P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
W20260807 03:40:02.024187 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 82177.2us
I20260807 03:40:01.912418 23243 raft_consensus.cc:504] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:02.027410 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bab68c1264fc8a2c4164e0f17f9b6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:40:01.937533 18249 leader_election.cc:302] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49; no voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:40:01.919119 23271 raft_consensus.cc:482] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:40:01.955624 18297 raft_consensus.cc:2454] T b2d369700aef4621b22ba058956d0894 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 7.
W20260807 03:40:01.996093 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 327465us
I20260807 03:40:01.903767 23207 raft_consensus.cc:504] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:01.904367 23264 consensus_queue.cc:1035] T 7285c5f72c354324b78146f4ed947cad P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20260807 03:40:01.936025 22980 raft_consensus.cc:686] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [term 10 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
W20260807 03:40:01.956079 23477 log.cc:924] Time spent T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.995s user 0.009s sys 0.000s
I20260807 03:40:01.910012 23150 raft_consensus.cc:504] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:01.904511 23508 raft_consensus.cc:482] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:40:02.032835 18310 raft_consensus.cc:3037] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Advancing to term 9
I20260807 03:40:02.065307 18309 raft_consensus.cc:2396] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260807 03:40:01.984467 23534 raft_consensus.cc:686] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [term 6 LEADER]: Becoming Leader. State: Replica: 5c9b5062bdfa46af9396f1a85025a331, State: Running, Role: LEADER
I20260807 03:40:02.034822 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:40:01.932366 23430 raft_consensus.cc:504] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:02.036484 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6399a0e5977a46c8b09c11f11af795e8" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:40:02.031975 22984 raft_consensus.cc:2726] T d8d26ed9a1e340ce8c2d127ac7fbf7f7 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260807 03:40:02.056803 18305 raft_consensus.cc:1261] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Refusing update from remote peer 5c9b5062bdfa46af9396f1a85025a331: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260807 03:40:02.064080 18295 raft_consensus.cc:2396] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c4bceb836db4536a0f255fdf86abc02 for term 11 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20260807 03:40:02.024649 23145 raft_consensus.cc:482] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 03:40:02.031827 23536 log.cc:924] Time spent T 0890f92b53f04149998f1f1c4e7ba60d P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.123s user 0.001s sys 0.000s
I20260807 03:40:01.947945 23010 raft_consensus.cc:504] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:02.036459 23271 raft_consensus.cc:3037] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Advancing to term 5
I20260807 03:40:02.029505 23426 leader_election.cc:288] T 60437eed8852484c88e84993e376670e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:40:01.947851 23460 consensus_queue.cc:1046] T 28ccf467b7d24422ba092003f6b06616 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.115s
W20260807 03:40:02.079274 23540 log.cc:924] Time spent T 00000000000000000000000000000000 P 8da2065a77024f35bdfc0dc3adb83f69: Append to log took a long time: real 0.051s user 0.001s sys 0.000s
I20260807 03:40:02.023929 23045 consensus_queue.cc:1035] T 4ba8413916634e039111c838fbe1e587 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20260807 03:40:01.959483 23201 raft_consensus.cc:504] T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
W20260807 03:40:02.107353 18249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.130:37057, user_credentials={real_user=slave}} blocked reactor thread for 169915us
I20260807 03:40:02.120147 18193 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "96140f11b8e74293b333cd7e760c3959"
dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
mode: GRACEFUL
new_leader_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
from {username='slave'} at 127.0.0.1:34758
I20260807 03:40:02.099285 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:40:02.121369 18311 raft_consensus.cc:2454] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 5.
W20260807 03:40:02.112498 19239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 326055us
I20260807 03:40:02.186108 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae0265e0707432980672f7cc955c571" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:40:02.026021 23427 raft_consensus.cc:504] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:02.114127 23426 raft_consensus.cc:482] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 03:40:02.267974 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 240278us
I20260807 03:40:02.079936 23508 raft_consensus.cc:504] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:02.080749 23145 raft_consensus.cc:504] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:02.113279 23480 consensus_queue.cc:237] T 23b69f685f5d40edbdd2981d75bc2ad2 P d9716e2bf06442f5b3b0e81087c53c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:40:02.664682 18300 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.568s user 0.009s sys 0.006s
W20260807 03:40:02.695441 18300 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.612s user 0.009s sys 0.009s
I20260807 03:40:02.696233 18300 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57268 (request call id 2224) took 1082 ms. Trace:
I20260807 03:40:02.467424 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ca0223f47b46fd955704eba5259370" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:40:02.328974 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:40:02.576009 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5db05c008b404288067417c3f100d2" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:40:02.699851 18311 raft_consensus.cc:2454] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 7.
I20260807 03:40:02.370884 18187 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44660 (request call id 2641) took 2095 ms. Trace:
I20260807 03:40:02.670709 18246 leader_election.cc:398] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 election: Vote denied by peer 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381) with higher term. Message: Invalid argument: T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 8. Current term is 9.
I20260807 03:40:02.701912 18246 leader_election.cc:401] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260807 03:40:02.289065 18248 leader_election.cc:302] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
W20260807 03:40:02.529321 23535 log.cc:924] Time spent T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.480s user 0.001s sys 0.000s
I20260807 03:40:02.246711 23207 leader_election.cc:288] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
W20260807 03:40:02.354374 18293 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.694s user 0.040s sys 0.076s
W20260807 03:40:02.644475 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 458550us
I20260807 03:40:02.699795 18305 raft_consensus.cc:2454] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 4.
W20260807 03:40:02.716293 18246 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.132:41381, user_credentials={real_user=slave}} blocked reactor thread for 68261.7us
I20260807 03:40:02.721001 18246 leader_election.cc:302] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:40:02.722393 18246 leader_election.cc:302] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c9b5062bdfa46af9396f1a85025a331, 66a36e1ac8e945e893668e282d394c49; no voters:
I20260807 03:40:02.601069 23243 leader_election.cc:288] T 4a8b888de6f046a88c6832c234e5de79 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:40:02.723524 23243 raft_consensus.cc:482] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
W20260807 03:40:02.725025 23343 consensus_meta.cc:287] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.787s user 0.010s sys 0.001s
I20260807 03:40:02.725438 23343 raft_consensus.cc:504] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:40:02.726009 23013 consensus_meta.cc:287] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d: Time spent flushing consensus metadata: real 1.171s user 0.011s sys 0.000s
I20260807 03:40:02.726490 23013 raft_consensus.cc:2726] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260807 03:40:02.727039 23013 raft_consensus.cc:482] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:40:02.727401 23343 leader_election.cc:288] T c8c916a6b2a8442c9a05c532603c5904 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 11 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
W20260807 03:40:02.623172 23539 log.cc:924] Time spent T 6b66966247c147308a4b8488557e0bdf P 3c4bceb836db4536a0f255fdf86abc02: Append to log took a long time: real 0.552s user 0.001s sys 0.000s
I20260807 03:40:02.727769 23508 leader_election.cc:288] T c702701ec4504375977f142ffc7f0be0 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:40:02.734378 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91555cf08ec8405c9d002ad0c11f8e5e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
W20260807 03:40:02.719529 18293 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.059s user 0.074s sys 0.120s
I20260807 03:40:02.267865 23132 raft_consensus.cc:2726] T 14dbf332d6394c66937326ef608dc8b5 P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260807 03:40:02.591969 23248 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.848s user 0.001s sys 0.006s
I20260807 03:40:02.518628 23460 raft_consensus.cc:482] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
W20260807 03:40:02.732669 18310 consensus_meta.cc:287] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49: Time spent flushing consensus metadata: real 0.653s user 0.010s sys 0.000s
W20260807 03:40:02.550067 23512 log.cc:924] Time spent T 2371c1d6174640aa9a9ee19732634a88 P d9716e2bf06442f5b3b0e81087c53c4d: Append to log took a long time: real 0.402s user 0.000s sys 0.001s
I20260807 03:40:02.283423 23430 leader_election.cc:288] T 922e1d3cde8f476ea7780363c6b8693c P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:40:02.448627 23284 leader_election.cc:288] T 4cd2b7ebeed04ac6907b51b064e94894 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:40:02.750021 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9b0210bce148c4a06eef3019b593c6" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:40:02.779503 18297 raft_consensus.cc:3037] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 11 FOLLOWER]: Advancing to term 12
I20260807 03:40:02.281682 23426 raft_consensus.cc:504] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:02.779907 18072 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33268 (request call id 2134) took 1985 ms. Trace:
I20260807 03:40:02.780364 18072 rpcz_store.cc:276] 0807 03:40:00.794911 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:40:00.984815 (+189904us) service_pool.cc:227] Handling call
0807 03:40:00.993603 (+ 8788us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:40:01.060519 (+ 66916us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:40:01.060565 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:40:01.091090 (+ 30525us) log.cc:842] Serialized 26 byte log entry
0807 03:40:01.392437 (+301347us) raft_consensus.cc:1631] Marking committed up to 6
0807 03:40:01.392510 (+ 73us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:40:01.392604 (+ 94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:40:02.779636 (+1387032us) raft_consensus.cc:1671] finished
0807 03:40:02.779662 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:40:02.779809 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10998,"raft.queue_time_us":68997,"raft.run_cpu_time_us":233,"raft.run_wall_time_us":41487,"thread_start_us":262639,"threads_started":1,"wal-append.queue_time_us":475708}
I20260807 03:40:02.782262 23426 leader_election.cc:288] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:40:02.577431 23467 leader_election.cc:288] T 7b26fcaa55344cf98fc76dd083788d00 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:40:02.757102 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60437eed8852484c88e84993e376670e" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:40:02.782745 18310 raft_consensus.cc:2454] T fe8bab68c1264fc8a2c4164e0f17f9b6 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 9.
I20260807 03:40:02.576238 23422 leader_election.cc:288] T c1476cce1786402ab984d773b26acb1b P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:40:02.701706 18187 rpcz_store.cc:276] 0807 03:40:00.265258 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:40:00.265374 (+ 116us) service_pool.cc:227] Handling call
0807 03:40:00.277579 (+ 12205us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:40:00.527760 (+250181us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 03:40:00.527797 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:40:00.544998 (+ 17201us) log.cc:842] Serialized 26 byte log entry
0807 03:40:00.782553 (+237555us) raft_consensus.cc:1631] Marking committed up to 5
0807 03:40:00.782605 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:40:00.784528 (+ 1923us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:40:01.676552 (+892024us) spinlock_profiling.cc:243] Waited 16.2 ms on lock 0x7b44000232cc. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f1a9a95 00007fe98f1aa4b2 00007fe991e52511 00007fe991e4de1a 00007fe991e41bee 00007fe991e40f36 00007fe991e40b58 00007fe991e4a551 00007fe991e4a509 00007fe991e4a499
0807 03:40:02.330297 (+653745us) raft_consensus.cc:1671] finished
0807 03:40:02.330321 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:40:02.360902 (+ 30581us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":83110,"raft.queue_time_us":117027,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":14797,"spinlock_wait_cycles":37196672,"thread_start_us":386684,"threads_started":2,"wal-append.queue_time_us":386259}
I20260807 03:40:02.533075 23150 leader_election.cc:288] T 1f41eb41bec8494c8a6708fbc44a22f1 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:40:02.599472 23010 leader_election.cc:288] T 4ba8413916634e039111c838fbe1e587 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:40:02.728331 23343 raft_consensus.cc:482] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:40:02.788686 23343 raft_consensus.cc:504] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:40:02.667752 19291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:40:02.279464 22980 consensus_queue.cc:237] T e8d6cc12bed1425985f0504451c5e057 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:02.802014 18297 raft_consensus.cc:2454] T 8a9b0210bce148c4a06eef3019b593c6 P 66a36e1ac8e945e893668e282d394c49 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4bceb836db4536a0f255fdf86abc02 in term 12.
I20260807 03:40:02.758419 18027 leader_election.cc:302] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
W20260807 03:40:02.763592 23417 consensus_meta.cc:287] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 1.199s user 0.013s sys 0.000s
I20260807 03:40:02.499876 23254 raft_consensus.cc:482] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:40:02.458407 23269 raft_consensus.cc:482] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:40:02.337359 23534 consensus_queue.cc:237] T 1b214306f53244f8b6ff2a1a9019abd1 P 5c9b5062bdfa46af9396f1a85025a331 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:40:02.679946 23427 leader_election.cc:288] T 64ca0223f47b46fd955704eba5259370 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057)
I20260807 03:40:02.448436 23432 leader_election.cc:288] T 4d3a25995c8341c0bf39232774e3a917 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:40:02.279574 23470 raft_consensus.cc:482] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:40:02.696663 18300 rpcz_store.cc:276] 0807 03:40:01.613550 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:40:01.613746 (+ 196us) service_pool.cc:227] Handling call
0807 03:40:01.656559 (+ 42813us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:40:01.656838 (+ 279us) raft_consensus.cc:1520] Early marking committed up to index 6
0807 03:40:01.938681 (+281843us) log.cc:842] Serialized 14 byte log entry
0807 03:40:02.695804 (+757123us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:40:02.695895 (+ 91us) raft_consensus.cc:1631] Marking committed up to 6
0807 03:40:02.695951 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:40:02.696017 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:40:02.696162 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5191,"prepare.queue_time_us":540735,"prepare.run_wall_time_us":45,"raft.queue_time_us":164454,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":11420,"spinlock_wait_cycles":107835776,"thread_start_us":977589,"threads_started":3,"wal-append.queue_time_us":169587}
I20260807 03:40:02.447724 22862 raft_consensus.cc:482] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:40:02.596274 23261 leader_election.cc:288] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
W20260807 03:40:02.836231 19239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 191035us
I20260807 03:40:02.611672 23567 consensus_queue.cc:1046] T 6399a0e5977a46c8b09c11f11af795e8 P 66a36e1ac8e945e893668e282d394c49 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.154s
I20260807 03:40:02.727487 23013 raft_consensus.cc:504] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:40:02.915337 23013 leader_election.cc:288] T 1b214306f53244f8b6ff2a1a9019abd1 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:40:02.915788 23013 raft_consensus.cc:482] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:40:02.916229 23013 raft_consensus.cc:504] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:02.916728 19290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b214306f53244f8b6ff2a1a9019abd1" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
W20260807 03:40:02.765578 23248 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.025s user 0.007s sys 0.011s
I20260807 03:40:02.917964 23248 raft_consensus.cc:686] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 LEADER]: Becoming Leader. State: Replica: 3c4bceb836db4536a0f255fdf86abc02, State: Running, Role: LEADER
W20260807 03:40:02.919023 23442 consensus_meta.cc:287] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 1.415s user 0.010s sys 0.001s
I20260807 03:40:02.918926 23248 consensus_queue.cc:237] T 3433ce141bc54c5395e00fe40cf9bb69 P 3c4bceb836db4536a0f255fdf86abc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:40:02.919461 23442 raft_consensus.cc:504] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:02.760102 23553 raft_consensus.cc:482] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
W20260807 03:40:02.867844 18246 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.132:41381, user_credentials={real_user=slave}} blocked reactor thread for 145536us
I20260807 03:40:02.723976 23243 raft_consensus.cc:504] T 7afbf00d2d944c18921cc32b7c48be16 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:02.827271 18305 raft_consensus.cc:1261] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Refusing update from remote peer 3c4bceb836db4536a0f255fdf86abc02: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
W20260807 03:40:02.887668 23411 consensus_meta.cc:287] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 1.307s user 0.004s sys 0.009s
I20260807 03:40:02.819861 23546 raft_consensus.cc:482] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 03:40:02.889484 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 131161us
I20260807 03:40:02.869329 23343 leader_election.cc:288] T 6b66966247c147308a4b8488557e0bdf P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381)
I20260807 03:40:02.904233 18187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:40:02.904791 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:40:02.866935 22897 raft_consensus.cc:482] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:40:02.905100 18188 raft_consensus.cc:1261] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: Refusing update from remote peer d9716e2bf06442f5b3b0e81087c53c4d: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 6. (term mismatch)
I20260807 03:40:02.870311 23551 raft_consensus.cc:482] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:40:02.908589 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c98c1d422654739af06653a512f7499" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:40:02.909298 23474 raft_consensus.cc:482] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:40:02.868348 23417 raft_consensus.cc:504] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:40:02.892199 23426 raft_consensus.cc:482] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:40:02.890266 23010 raft_consensus.cc:2726] T d0047935053942d480774d46b081709d P d9716e2bf06442f5b3b0e81087c53c4d [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260807 03:40:02.905330 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cf89cd640d48bcbbf376c2d01e79d4" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:40:02.821900 23460 raft_consensus.cc:504] T 79e3ecc1bf7d4ca9a7096616efc98585 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:03.047237 18293 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57268 (request call id 2221) took 1435 ms. Trace:
I20260807 03:40:03.048054 18305 pending_rounds.cc:76] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49: Aborting all ops after (but not including) 3
I20260807 03:40:03.048624 18305 pending_rounds.cc:97] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260807 03:40:03.048899 18305 raft_consensus.cc:2866] T e5ea193a7604411b81684ccff437bc0a P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260807 03:40:02.920619 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 631572us
I20260807 03:40:03.049363 18293 rpcz_store.cc:276] 0807 03:40:01.612667 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:40:01.612792 (+ 125us) service_pool.cc:227] Handling call
0807 03:40:01.613092 (+ 300us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:40:01.614846 (+ 1754us) raft_consensus.cc:1520] Early marking committed up to index 4
0807 03:40:02.762745 (+1147899us) log.cc:842] Serialized 14 byte log entry
0807 03:40:03.046832 (+284087us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:40:03.046906 (+ 74us) raft_consensus.cc:1631] Marking committed up to 4
0807 03:40:03.046967 (+ 61us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:40:03.047056 (+ 89us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:40:03.047180 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":465290,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"thread_start_us":1385656,"threads_started":2}
I20260807 03:40:02.866355 23207 raft_consensus.cc:2726] T 3433ce141bc54c5395e00fe40cf9bb69 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 5c9b5062bdfa46af9396f1a85025a331 (127.15.198.132:41381) with higher term. Message: Invalid argument: T 3433ce141bc54c5395e00fe40cf9bb69 P 5c9b5062bdfa46af9396f1a85025a331 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 for earlier term 8. Current term is 9.
I20260807 03:40:03.036749 18190 raft_consensus.cc:2454] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 7.
I20260807 03:40:03.063860 18073 raft_consensus.cc:1226] T 1f5db05c008b404288067417c3f100d2 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260807 03:40:02.894008 23150 raft_consensus.cc:2781] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260807 03:40:03.024360 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702701ec4504375977f142ffc7f0be0" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:40:02.972194 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e3bbfb1c74e1686eae36683a96314" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:40:02.896958 23422 raft_consensus.cc:2781] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 03:40:02.891935 23585 raft_consensus.cc:2781] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 03:40:03.074077 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:40:03.075219 18141 leader_election.cc:302] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:40:02.985931 23480 raft_consensus.cc:482] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:40:02.985154 18193 raft_consensus.cc:595] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 11 FOLLOWER]: Received request to transfer leadership to TS d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:40:03.055464 18069 raft_consensus.cc:1226] T 91555cf08ec8405c9d002ad0c11f8e5e P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Rejecting Update request from peer 66a36e1ac8e945e893668e282d394c49 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260807 03:40:02.971514 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9d935105d41829202c4c161ff09e3" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:40:03.058043 18027 leader_election.cc:302] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49
I20260807 03:40:02.988969 23442 leader_election.cc:288] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 11 election: Requested vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:40:02.996345 23467 raft_consensus.cc:482] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:40:03.096377 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:40:02.921452 18180 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50060 (request call id 243) took 1397 ms. Trace:
I20260807 03:40:03.073570 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:40:03.097332 18070 raft_consensus.cc:2454] T 4cd2b7ebeed04ac6907b51b064e94894 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 9.
I20260807 03:40:02.921013 23261 raft_consensus.cc:482] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 66a36e1ac8e945e893668e282d394c49)
I20260807 03:40:02.973328 23508 raft_consensus.cc:482] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:40:03.050981 23421 raft_consensus.cc:482] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:40:03.083401 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:40:03.035645 18188 pending_rounds.cc:76] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02: Aborting all ops after (but not including) 5
I20260807 03:40:03.091737 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3a25995c8341c0bf39232774e3a917" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:40:03.099213 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:40:03.085209 18185 raft_consensus.cc:2454] T c7a9d935105d41829202c4c161ff09e3 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 7.
I20260807 03:40:03.087838 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:40:03.100605 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbff0b8e97846ea9f567f377b986efc" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:40:03.100886 18141 leader_election.cc:302] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:40:03.101112 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d"
I20260807 03:40:03.101866 18079 raft_consensus.cc:3037] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 10 FOLLOWER]: Advancing to term 11
I20260807 03:40:03.102275 18141 leader_election.cc:302] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:40:03.103549 18188 pending_rounds.cc:97] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02: Aborting uncommitted NO_OP operation due to leader change: 7.6
I20260807 03:40:03.104245 18188 raft_consensus.cc:2866] T 14dbf332d6394c66937326ef608dc8b5 P 3c4bceb836db4536a0f255fdf86abc02 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 03:40:03.085144 18193 raft_consensus.cc:603] T 96140f11b8e74293b333cd7e760c3959 P 3c4bceb836db4536a0f255fdf86abc02 [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 03:40:03.093858 18080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ca0223f47b46fd955704eba5259370" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:40:02.906488 22862 raft_consensus.cc:504] T 8c98c1d422654739af06653a512f7499 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:40:03.096140 18246 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.132:41381, user_credentials={real_user=slave}} blocked reactor thread for 59659.6us
I20260807 03:40:03.057137 23021 raft_consensus.cc:2781] T 349190ae153f46c9a99c29df20734d3f P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 03:40:03.099939 18072 raft_consensus.cc:2454] T 1f41eb41bec8494c8a6708fbc44a22f1 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 9.
I20260807 03:40:03.057188 22897 raft_consensus.cc:504] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:03.068630 23150 raft_consensus.cc:482] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:40:03.070422 23567 raft_consensus.cc:482] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:40:03.090087 23432 raft_consensus.cc:482] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5c9b5062bdfa46af9396f1a85025a331)
I20260807 03:40:03.126526 18141 leader_election.cc:302] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02; no voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:40:03.107997 18190 raft_consensus.cc:2379] T 1dbff0b8e97846ea9f567f377b986efc P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66a36e1ac8e945e893668e282d394c49 in current term 10: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:40:03.108348 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:40:03.108660 18071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26fcaa55344cf98fc76dd083788d00" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:40:03.097096 18180 rpcz_store.cc:276] 0807 03:40:01.523587 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:40:01.525265 (+ 1678us) service_pool.cc:227] Handling call
0807 03:40:01.525623 (+ 358us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 03:40:01.538760 (+ 13137us) raft_consensus.cc:1520] Early marking committed up to index 8
0807 03:40:01.538797 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 03:40:01.560543 (+ 21746us) log.cc:842] Serialized 26 byte log entry
0807 03:40:01.781662 (+221119us) raft_consensus.cc:1631] Marking committed up to 8
0807 03:40:01.781718 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:40:01.783755 (+ 2037us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 03:40:02.921238 (+1137483us) raft_consensus.cc:1671] finished
0807 03:40:02.921261 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:40:02.921382 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1415,"raft.queue_time_us":9993,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":7757,"thread_start_us":202906,"threads_started":1,"wal-append.queue_time_us":449895}
I20260807 03:40:02.905893 23470 raft_consensus.cc:504] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:03.105576 18080 raft_consensus.cc:2454] T 64ca0223f47b46fd955704eba5259370 P d9716e2bf06442f5b3b0e81087c53c4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 3.
I20260807 03:40:03.074599 23585 raft_consensus.cc:482] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Starting leader election (detected failure of leader d9716e2bf06442f5b3b0e81087c53c4d)
I20260807 03:40:03.111547 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67de6ea5223f4eefa2edd875f9dfd1bc" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:40:03.114466 18293 raft_consensus.cc:1226] T 1b214306f53244f8b6ff2a1a9019abd1 P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Rejecting Update request from peer d9716e2bf06442f5b3b0e81087c53c4d for earlier term 5. Current term is 6. Ops: []
I20260807 03:40:03.112480 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:40:03.149559 18185 raft_consensus.cc:2379] T 67de6ea5223f4eefa2edd875f9dfd1bc P 3c4bceb836db4536a0f255fdf86abc02 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9716e2bf06442f5b3b0e81087c53c4d in current term 14: Already voted for candidate 3c4bceb836db4536a0f255fdf86abc02 in this term.
I20260807 03:40:03.098428 23544 raft_consensus.cc:2726] T 4bd704177fca4d3bbd1b9d39a5f3ec01 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260807 03:40:03.049830 23474 raft_consensus.cc:504] T 9a34af1cebc14f92bad043893aa1f964 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:40:03.118119 18312 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.548s user 0.005s sys 0.013s
W20260807 03:40:03.151192 18312 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.581s user 0.005s sys 0.013s
I20260807 03:40:03.046873 23411 raft_consensus.cc:504] T 64ca0223f47b46fd955704eba5259370 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:02.928004 23269 raft_consensus.cc:504] T 79e3ecc1bf7d4ca9a7096616efc98585 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:40:03.123436 18027 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 73519us
I20260807 03:40:03.096235 23013 leader_election.cc:288] T 7afbf00d2d944c18921cc32b7c48be16 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3c4bceb836db4536a0f255fdf86abc02 (127.15.198.130:37057), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:40:03.091060 23422 raft_consensus.cc:686] T 071a939ead594d38851e3b554bfc4d24 P 66a36e1ac8e945e893668e282d394c49 [term 5 LEADER]: Becoming Leader. State: Replica: 66a36e1ac8e945e893668e282d394c49, State: Running, Role: LEADER
I20260807 03:40:03.057338 23426 raft_consensus.cc:504] T d0047935053942d480774d46b081709d P 3c4bceb836db4536a0f255fdf86abc02 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
W20260807 03:40:03.154955 18248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.198.129:36163, user_credentials={real_user=slave}} blocked reactor thread for 49974.6us
I20260807 03:40:03.051000 23546 raft_consensus.cc:504] T 6129789267024fac8a337070c10a472a P 3c4bceb836db4536a0f255fdf86abc02 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
I20260807 03:40:03.155509 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"
I20260807 03:40:03.089486 23207 raft_consensus.cc:482] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3c4bceb836db4536a0f255fdf86abc02)
I20260807 03:40:03.127521 18184 raft_consensus.cc:2454] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 6.
I20260807 03:40:03.129719 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" is_pre_election: true
I20260807 03:40:02.978346 23254 raft_consensus.cc:504] T 23b69f685f5d40edbdd2981d75bc2ad2 P 3c4bceb836db4536a0f255fdf86abc02 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:03.147990 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d416e1015a2644958b0c8f9147d403a2" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:40:03.083062 23480 raft_consensus.cc:504] T e5ea193a7604411b81684ccff437bc0a P d9716e2bf06442f5b3b0e81087c53c4d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:03.050251 23551 raft_consensus.cc:504] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:03.151702 23585 raft_consensus.cc:3037] T 303e3bbfb1c74e1686eae36683a96314 P 66a36e1ac8e945e893668e282d394c49 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:40:03.109758 23442 raft_consensus.cc:2781] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 03:40:03.109318 23010 raft_consensus.cc:2726] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260807 03:40:03.124262 23271 consensus_meta.cc:287] T b58a8dc36ec94e0c8877bdf0f901598a P 3c4bceb836db4536a0f255fdf86abc02: Time spent flushing consensus metadata: real 1.025s user 0.011s sys 0.001s
I20260807 03:40:03.152478 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:40:03.131266 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccf467b7d24422ba092003f6b06616" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:40:03.140251 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b888de6f046a88c6832c234e5de79" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:40:03.121930 23150 raft_consensus.cc:3037] T 1dbff0b8e97846ea9f567f377b986efc P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Advancing to term 10
I20260807 03:40:03.115584 23564 raft_consensus.cc:482] T d0047935053942d480774d46b081709d P 66a36e1ac8e945e893668e282d394c49 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:40:03.110530 23421 raft_consensus.cc:504] T c3de9873c6144d36b33a1a666691869a P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:03.168476 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1476cce1786402ab984d773b26acb1b" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:40:03.158167 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f41eb41bec8494c8a6708fbc44a22f1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:40:03.045679 23553 raft_consensus.cc:504] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } attrs { replace: true } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: NON_VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: true } }
I20260807 03:40:03.175700 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0890f92b53f04149998f1f1c4e7ba60d" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:40:03.174778 18183 raft_consensus.cc:2454] T c702701ec4504375977f142ffc7f0be0 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 7.
I20260807 03:40:03.108557 23508 raft_consensus.cc:504] T bb66518c65e4495e881ce860253d0801 P d9716e2bf06442f5b3b0e81087c53c4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:03.104367 23467 raft_consensus.cc:504] T 13bfb55b1d7f43dc9fb01c519c175ccf P 66a36e1ac8e945e893668e282d394c49 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:03.178028 18312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57268 (request call id 2228) took 1550 ms. Trace:
I20260807 03:40:03.141804 18079 raft_consensus.cc:2454] T c8c916a6b2a8442c9a05c532603c5904 P d9716e2bf06442f5b3b0e81087c53c4d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 11.
I20260807 03:40:03.152673 23145 leader_election.cc:288] T 13bfb55b1d7f43dc9fb01c519c175ccf P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d9716e2bf06442f5b3b0e81087c53c4d (127.15.198.129:36163), 66a36e1ac8e945e893668e282d394c49 (127.15.198.131:33021)
I20260807 03:40:03.154018 23582 raft_consensus.cc:482] T 6399a0e5977a46c8b09c11f11af795e8 P d9716e2bf06442f5b3b0e81087c53c4d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:40:03.187196 18184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c916a6b2a8442c9a05c532603c5904" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:40:03.195148 18082 raft_consensus.cc:2454] T 4a8b888de6f046a88c6832c234e5de79 P d9716e2bf06442f5b3b0e81087c53c4d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 9.
I20260807 03:40:03.175673 22441 raft_consensus.cc:2726] T 7b26fcaa55344cf98fc76dd083788d00 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 03:40:03.153215 23609 raft_consensus.cc:2726] T 4ae0265e0707432980672f7cc955c571 P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260807 03:40:03.197214 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:40:03.191740 18189 raft_consensus.cc:3037] T 28ccf467b7d24422ba092003f6b06616 P 3c4bceb836db4536a0f255fdf86abc02 [term 7 FOLLOWER]: Advancing to term 8
I20260807 03:40:03.232486 18140 leader_election.cc:302] T 91555cf08ec8405c9d002ad0c11f8e5e P 3c4bceb836db4536a0f255fdf86abc02 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, d9716e2bf06442f5b3b0e81087c53c4d; no voters:
I20260807 03:40:03.112557 23261 raft_consensus.cc:504] T 4ba8413916634e039111c838fbe1e587 P 3c4bceb836db4536a0f255fdf86abc02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:03.177908 23207 raft_consensus.cc:504] T 8c98c1d422654739af06653a512f7499 P 66a36e1ac8e945e893668e282d394c49 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:03.197570 18188 raft_consensus.cc:2454] T 4a8b888de6f046a88c6832c234e5de79 P 3c4bceb836db4536a0f255fdf86abc02 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 9.
I20260807 03:40:03.148690 23432 raft_consensus.cc:504] T a3dc635ee56b4559a87f5955bb83235a P 66a36e1ac8e945e893668e282d394c49 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } } peers { permanent_uuid: "5c9b5062bdfa46af9396f1a85025a331" member_type: VOTER last_known_addr { host: "127.15.198.132" port: 41381 } attrs { promote: false } }
W20260807 03:40:03.198534 18141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.198.131:33021, user_credentials={real_user=slave}} blocked reactor thread for 66753.9us
I20260807 03:40:03.199786 18312 rpcz_store.cc:276] 0807 03:40:01.627361 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 03:40:01.684934 (+ 57573us) service_pool.cc:227] Handling call
0807 03:40:02.187078 (+502144us) spinlock_profiling.cc:243] Waited 6.9 ms on lock 0x7b58000e0498. stack: 00007fe98f265033 00007fe98f264f30 00007fe98ebc3534 00007fe9988cf8b4 00007fe9988cf849 00007fe98f0dbf11 00007fe98f25f851 00007fe98f25f702 00007fe99814fb09 00007fe99708b730 00007fe99715179b 00007fe997151821 00007fe9971152b0 00007fe9971007f9 00007fe9971002a2 00007fe9907e6684
0807 03:40:02.187307 (+ 229us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 03:40:02.187561 (+ 254us) raft_consensus.cc:1520] Early marking committed up to index 6
0807 03:40:02.273623 (+ 86062us) log.cc:842] Serialized 14 byte log entry
0807 03:40:03.158777 (+885154us) log.cc:842] Serialized 14 byte log entry
0807 03:40:03.177713 (+ 18936us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 03:40:03.177763 (+ 50us) raft_consensus.cc:1631] Marking committed up to 6
0807 03:40:03.177802 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 03:40:03.177862 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 03:40:03.177979 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":322622,"prepare.queue_time_us":753977,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":75,"raft.queue_time_us":370594,"raft.run_cpu_time_us":278,"raft.run_wall_time_us":38732,"spinlock_wait_cycles":15872896,"thread_start_us":684343,"threads_started":2,"wal-append.queue_time_us":486581}
I20260807 03:40:03.238881 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:40:03.204453 18185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca2374bfc1e4584bd24f81b2ab5d0f1" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02" is_pre_election: true
I20260807 03:40:03.204473 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:40:03.205117 18027 leader_election.cc:302] T 96140f11b8e74293b333cd7e760c3959 P d9716e2bf06442f5b3b0e81087c53c4d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66a36e1ac8e945e893668e282d394c49, d9716e2bf06442f5b3b0e81087c53c4d; no voters: 3c4bceb836db4536a0f255fdf86abc02
I20260807 03:40:03.130641 23567 raft_consensus.cc:504] T c3de9873c6144d36b33a1a666691869a P 66a36e1ac8e945e893668e282d394c49 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" member_type: VOTER last_known_addr { host: "127.15.198.129" port: 36163 } } peers { permanent_uuid: "3c4bceb836db4536a0f255fdf86abc02" member_type: VOTER last_known_addr { host: "127.15.198.130" port: 37057 } } peers { permanent_uuid: "66a36e1ac8e945e893668e282d394c49" member_type: VOTER last_known_addr { host: "127.15.198.131" port: 33021 } }
I20260807 03:40:03.206992 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd2b7ebeed04ac6907b51b064e94894" candidate_uuid: "d9716e2bf06442f5b3b0e81087c53c4d" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49" is_pre_election: true
I20260807 03:40:03.186764 23442 raft_consensus.cc:482] T 8ca2374bfc1e4584bd24f81b2ab5d0f1 P 3c4bceb836db4536a0f255fdf86abc02 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:40:03.214061 18184 raft_consensus.cc:3037] T c8c916a6b2a8442c9a05c532603c5904 P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Advancing to term 11
I20260807 03:40:03.215914 18249 leader_election.cc:302] T c7a9d935105d41829202c4c161ff09e3 P 66a36e1ac8e945e893668e282d394c49 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c4bceb836db4536a0f255fdf86abc02, 66a36e1ac8e945e893668e282d394c49; no voters: d9716e2bf06442f5b3b0e81087c53c4d
I20260807 03:40:03.233946 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3433ce141bc54c5395e00fe40cf9bb69" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3c4bceb836db4536a0f255fdf86abc02"
I20260807 03:40:03.213135 19291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b66966247c147308a4b8488557e0bdf" candidate_uuid: "66a36e1ac8e945e893668e282d394c49" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "5c9b5062bdfa46af9396f1a85025a331" is_pre_election: true
I20260807 03:40:03.234545 18187 raft_consensus.cc:2454] T d416e1015a2644958b0c8f9147d403a2 P 3c4bceb836db4536a0f255fdf86abc02 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9716e2bf06442f5b3b0e81087c53c4d in term 6.
I20260807 03:40:03.196071 18190 raft_consensus.cc:2454] T c1476cce1786402ab984d773b26acb1b P 3c4bceb836db4536a0f255fdf86abc02 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66a36e1ac8e945e893668e282d394c49 in term 10.
I20260807 03:40:03.239576 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dbf332d6394c66937326ef608dc8b5" candidate_uuid: "3c4bceb836db4536a0f255fdf86abc02" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "66a36e1ac8e945e893668e282d394c49"